The availability of FPGAs in cloud data centers offers rapid, on-demand access to reconfigurable hardware compute resources that users can adapt to their own needs. However, the low-level access to the FPGA hardware and associated resources such as the PCIe bus, SSD drives, or DRAM modules also opens up threats of malicious attackers uploading designs that are able to infer information about other users or about the cloud infrastructure itself. In particular, this work presents a new, fast PCIe-contention-based channel that is able to transmit data between FPGA-accelerated virtual machines (VMs) by modulating the PCIe bus usage. This channel further works with different operating systems and achieves bandwidths reaching 20 kbps with 99% accuracy. This is the first cross-FPGA covert channel demonstrated on commercial clouds and has a bandwidth which is over 2000× larger than prior voltage- or temperature-based cross-board attacks. This article further demonstrates that the PCIe receivers are able to not just receive covert transmissions, but can also perform fine-grained monitoring of the PCIe bus, including detecting when co-located VMs are initialized, even prior to their associated FPGAs being used. Moreover, the proposed mechanism can be used to infer the activities of other users, or even slow down the programming of the co-located FPGAs as well as other data transfers between the host and the FPGA. Beyond leaking information across different virtual machines, the ability to monitor the PCIe bandwidth over hours or days can be used to estimate the data center utilization and map the behavior of the other users. The article also introduces further novel threats in FPGA-accelerated instances, including contention due to network traffic, contention due to shared NVMe SSDs, as well as thermal monitoring to identify FPGA co-location using the DRAM modules attached to the FPGA boards. This is the first work to demonstrate that it is possible to break the separation of privilege in FPGA-accelerated cloud environments, and highlights that defenses for public clouds using FPGAs need to consider PCIe, SSD, and DRAM resources as part of the attack surface that should be protected.

In complex FPGA designs, implementations of algorithms and protocols from third-party sources are common. However, the monolithic nature of FPGAs means that all sub-circuits share common on-chip infrastructure, such as routing resources. This presents an attack vector for all FPGAs that contain designs from multiple vendors, especially for FPGAs used in multi-tenant cloud environments, or integrated into multi-core processors: hardware imperfections can be used to infer high-level state and break security guarantees. In this paper, we demonstrate how “long” routing wires present can be used to for covert communication between disconnected cores, or by a malicious core to exfiltrate secrets. The information leakage is measurable for both static and dynamic signals, and that it can be detected using small on-board circuits. In our prototype we achieved 6 kbps bandwidth and 99.9% accuracy, and a side channel which can recover signals kept constant for only 128 cycles, with an accuracy of more than 98.4 %.

This paper introduces a non-invasive, low-cost, and trace-free capacitor freezing attack on DC/DC converters, which uses off-the-shelf electronics cooling sprays to rapidly decrease the capacitors' temperature. Due to the temperature drop, the converters are unable to maintain their specified output voltages, and their transient behavior changes. When the attack has finished, the temperature of the capacitors quickly returns to normal, while any evidence of the attack vanishes.

Public cloud infrastructures allow for easy, on-demand access to FPGA resources. However, the low-level, direct access to the FPGA hardware exposes the infrastructure providers to new types of attacks. Prior work has shown that it is possible to uniquely identify the underlying hardware by creating fingerprints of the different FPGA instances that users rent from a cloud provider, but such work was not able to actually map the cloud FPGA infrastructure itself. Meanwhile, this paper demonstrates that it is possible to reverse-engineer the co-location of FPGA boards inside a cloud FPGA server using PCIe contention. Specifically, this work deduces the Non-Uniform Memory Access (NUMA) locality of FPGA boards within a server by analyzing their mutual PCIe contention during simultaneous use of the PCIe bus. In addition, experiments conducted in data centers located in several geographic regions and repeated at different times are used to calculate the probability that cloud providers allocate FPGA boards co-located in the same server to a user. This paper thus shows that it is possible to map cloud FPGA infrastructures, and learn how FPGA instances are physically co-located within a server. Consequently, this paper also highlights the importance of mitigating these novel avenues for reverse-engineering and mapping of cloud FPGA setups, as they can reveal insights about the cloud infrastructure itself, or assist other single- and multi-tenant attacks.

Ring Oscillators (ROs) are fundamental primitives that are used as building blocks in many other types of circuits. This paper presents an in-depth characterization of ring oscillators which leverage the IOBUF primitive found in modern Xilinx FPGAs. This work first analyzes the impact of the drive strength and slew rate attributes of the IOBUFs on the ROs, and also characterizes the impacts of external temperature, internal voltage, and external voltage fluctuations on the frequency of the proposed ROs. This work further demonstrates that IOBUF-based ROs can detect whether electrical connections to the IOBUF pins have changed, including whether the DRAM module has been physically removed. Finally, the proposed ROs can be realized on cloud FPGAs, bypassing the restrictions that some cloud providers impose on combinatorial loops, and thus presenting a new security threat to remote FPGAs.

In recent years, multiple public cloud FPGA providers have emerged, increasing interest in FPGA acceleration of cryptographic, bioinformatic, financial, and machine learning algorithms. To help understand the security of the cloud FPGA infrastructures, this paper focuses on a fundamental question of understanding what an adversary can learn about the cloud FPGA infrastructure itself, without attacking it or damaging it. In particular, this work explores how unique features of FPGAs can be exploited to instantiate Physical Unclonable Functions (PUFs) that can distinguish between otherwise-identical FPGA boards. This paper specifically introduces the first method for identifying cloud FPGA instances by extracting a unique and stable FPGA fingerprint based on PUFs measured from the FPGA boards' DRAM modules. Experiments conducted on the Amazon Web Services (AWS) cloud reveal the probability of renting the same physical board more than once. Moreover, the experimental results show that hardware is not shared among f1.2xlarge, f1.4xlarge, and f1.16xlarge instance types. As the approach used does not violate any restrictions currently placed by Amazon, this paper also presents a set of defense mechanisms that can be added to existing countermeasures to mitigate users' attempts to fingerprint cloud FPGA infrastructures.

Information leakage in FPGAs poses a danger whenever multiple users share the reconfigurable fabric, for example in multi-tenant Cloud FPGAs, or whenever a potentially malicious IP module is synthesized within a single user's design on an FPGA. In such scenarios, capacitive crosstalk between so-called long routing wires has been previously shown to be a security vulnerability in both Xilinx and Intel FPGAs. Specifically, both static and dynamic values on long wires have been demonstrated to affect the delays of the adjacent long wires, and such delay changes have been exploited to steal sensitive information such as bits of cryptographic keys. While long-wire leakage is now well-understood and can be defended against, this work presents two other, new types of information leaks that pose similar risks, but which have not been studied in the past, and for which existing defenses do not work. First, this paper shows that other types of routing resources (namely medium wires) are also vulnerable to crosstalk, with changes in their delays also measurable fully on-chip. Second, this work introduces a novel source of information leaks that originates from logic elements within the FPGA Configurable Logic Blocks (CLBs) and is likely not the result of the capacitive crosstalk effects investigated in prior work. To understand the potential impact of the two new leakage sources, this paper experimentally characterizes and compares them in four families of Xilinx FPGAs, and discusses potential countermeasures in the context of existing attacks and defenses.

Field-Programmable Gate Arrays (FPGAs) are versatile, reconfigurable integrated circuits that can be used as hardware accelerators to process highly-sensitive data. Leaking this data and associated cryptographic keys, however, can undermine a system's security. To prevent potentially unintentional interactions that could break separation of privilege between different data center tenants, FPGAs in cloud environments are currently dedicated on a per-user basis. Nevertheless, while the FPGAs themselves are not shared among different users, other parts of the data center infrastructure are. This paper specifically shows for the first time that powering FPGAs, CPUs, and GPUs through the same power supply unit (PSU) can be exploited in FPGA-to-FPGA, CPU-to-FPGA, and GPU-to-FPGA covert channels between independent boards. These covert channels can operate remotely, without the need for physical access to, or modifications of, the boards. To demonstrate the attacks, this paper uses a novel combination of "sensing" and "stressing" ring oscillators as receivers on the sink FPGA. Further, ring oscillators are used as transmitters on the source FPGA. The transmitting and receiving circuits are used to determine the presence of the leakage on off-the-shelf Xilinx boards containing Artix 7 and Kintex 7 FPGA chips. Experiments are conducted with PSUs by two vendors, as well as CPUs and GPUs of different generations. Moreover, different sizes and types of ring oscillators are also tested. In addition, this work discusses potential countermeasures to mitigate the impact of the cross-board leakage. The results of this paper highlight the dangers of shared power supply units in local and cloud FPGAs, and therefore a fundamental need to re-think FPGA security for shared infrastructures.

Recent investigations into FPGA routing resources have shown that long wires in FPGAs leak information about their state in a way which can be measured using ring oscillators. Although in many cases this leakage does not pose a security threat, the possibility of multi-tenant use of FPGA resources invites potential side-and covert-channel attacks exploiting long wire leakage. However, prior work has ignored the realities of cloud environments, which may pose restrictions on the generated bitstreams, such as disallowing combinatorial loops. In this paper, we first demonstrate that the long wire leakage phenomenon persists even in the high-end Virtex UltraScale+ FPGA family. We then evaluate two ring oscillator designs that overcome combinatorial loop restrictions employed by cloud FPGA providers. We experimentally measure the long wire leakage of Virtex UltraScale+ FPGAs in the lab as well as in the Amazon and Huawei FPGA clouds. We show that the two new ring oscillator designs provide almost-identical estimates for the strength of the leakage as traditional ring oscillators, allowing us to measure femtosecond-scale changes in the delays of the long wires. We finally present a set of defense mechanisms that can prevent the new ring oscillator designs from being instantiated in the cloud and the long wire leakage from being exploited.

Field-Programmable Gate Arrays (FPGAs) are becoming increasingly available via commercial cloud providers, which currently allocate devices on a per-user basis. As the underlying hardware is often underutilized, several proposals for multi-tenant use of FPGA resources have been brought forth, along with some initial work on security attacks in this setting. Simultaneously, high-end FPGAs are being produced with 2.5D integration of multiple distinct dies, called Super Logic Regions (SLRs), onto the same chip. Although one might expect that physical separation of logic onto separate dies could prevent multi-tenant attacks, this paper demonstrates for the first time that cross-SLR information leaks based on sensing voltage changes within the FPGA chip are possible, without physical access to or modification of the boards. The cross-SLR covert channel is characterized analytically and experimentally on five Xilinx Virtex UltraScale+ FPGAs, both locally and on the Amazon and Huawei clouds. Several configurations of the source transmitters and the sink receivers are tested, including their locations, types, and sizes. The power-based channel is shown to have a bandwidth upwards of 4.6 Mbps and accuracy of over 97.6%. Consequently, as physical separation of tenants onto separate dies (SLRs) is an insufficient countermeasure against information leaks, hardware-level architectural improvements are necessary to make secure multi-tenant FPGAs on shared clouds a reality.

In complex FPGA designs, implementations of algorithms and protocols from third-party sources are common. However, the monolithic nature of FPGAs means that all sub-circuits share common on-chip infrastructure, such as routing resources. This presents an attack vector for all FPGAs that contain designs from multiple vendors, especially for FPGAs used in multi-tenant cloud environments, or integrated into multi-core processors. In this article, we show that “long” routing wires present a new source of information leakage on FPGAs, by influencing the delay of adjacent long wires. We show that the effect is measurable for both static and dynamic signals and that it can be detected using small on-board circuits. We characterize the channel in detail and show that it is measurable even when multiple competing circuits (including multiple long-wire transmitters) are present and can be replicated on different generations and families of Xilinx devices (Virtex 5, Virtex 6, Artix 7, and Spartan 7). We exploit the leakage to create a covert channel with 6kbps of bandwidth and 99.9% accuracy, and a side channel, which can recover signals kept constant for only 1.3sμs, with an accuracy of more than 98.4%. Finally, we propose countermeasures to reduce the impact of this leakage. 1

Field-Programmable Gate Arrays (FPGAs) are integrated circuits that implement reconfigurable hardware. They are used in modern systems, creating specialized, highly-optimized integrated circuits without the need to design and manufacture dedicated chips. As the capacity of FPGAs grows, it is increasingly common for designers to incorporate implementations of algorithms and protocols from a range of third-party sources. The monolithic nature of FPGAs means that all on-chip circuits, including third party black-box designs, must share common on-chip infrastructure, such as routing resources. In this paper, we observe that a "long" routing wire carrying a logical 1 reduces the propagation delay of other adjacent but unconnected long wires in the FPGA interconnect, thereby leaking information about its state. We exploit this effect and propose a communication channel that can be used for both covert transmissions between circuits, and for exfiltration of secrets from the chip. We show that the effect is measurable for both static and dynamic signals, and that it can be detected using very small on-board circuits. In our prototype, we are able to correctly infer the logical state of an adjacent long wire over 99% of the time, even without error correction, and for signals that are maintained for as little as 82us. Using a Manchester encoding scheme, our channel bandwidth is as high as 6kbps. We characterize the channel in detail and show that it is measurable even when multiple competing circuits are present and can be replicated on different generations and families of Xilinx devices (Virtex 5, Virtex 6, and Artix 7). Finally, we propose countermeasures that can be deployed by systems and tools designers to reduce the impact of this information leakage.

This paper presents the first distributed triangle listing algorithm with provable CPU, I/O, Memory, and Network bounds. Finding all triangles (3-cliques) in a graph has numerous applications for density and connectivity metrics, but the majority of existing algorithms for massive graphs are sequential, while distributed versions of algorithms do not guarantee their CPU, I/O, Memory, or Network requirements. Our Parallel and Distributed Triangle Listing (PDTL) framework focuses on efficient external-memory access in distributed environments instead of fitting sub graphs into memory. It works by performing efficient orientation and load-balancing steps, and replicating graphs across machines by using an extended version of Hu et al.'s Massive Graph Triangulation algorithm. PDTL suits a variety of computational environments, from single-core machines to high-end clusters, and computes the exact triangle count on graphs of over 6B edges and 1B vertices (e.g. Yahoo graphs), outperforming and using fewer resources than the state-of-the-art systems Power Graph, OPT, and PATRIC by 2x to 4x. Our approach thus highlights the importance of I/O in a distributed environment.