Network-on-chip (NoC) is a critical shared resource in scalable multicore processors; however, it is well-known that shared resources can lead to side-channel attacks. In this work, we demonstrate how contention for on-chip bandwidth in GPUs can lead to fine-grain information leakage and enable side-channel attacks. As a case study, we demonstrate how RSA key bit information can be leaked on a real GPU. We also describe how interconnect characteristics from the side-channel or an interconnect-gram can be used to fingerprint kernels executing on the GPU. To defend against such fine-grain side-channel attack, we propose secure arbitration that prevents information leakage while minimizing performance impact during normal execution. In particular, we present a novel ghost arbitration that prevents interconnect contention from being leveraged to leak information by keeping track of “ghost” requests or requests when other nodes receive free arbitration to enable least-recently-used priority. However, if the attacker reverse engineers the arbitration, a naive implementation of ghost arbitration can still lead to information leakage. Thus, we propose a weighted ghost arbitration that exploits “malicious” communication patterns to prevent information leakage with minimal loss in performance. Compared to previously proposed arbitration that is secure (e.g., strict time-division multiplexing), ghost arbitration is able to improve performance by up to$4\times$•

As GPUs are becoming widely deployed in the cloud infrastructure to support different application domains, the security concerns of GPUs are becoming increasingly important. In particular, the support for multiprogramming in modern GPUs has led to new vulnerabilities since multiple kernels in a GPU can be executed at the same time. In this work, we propose a new microarchitectural timing covert channel for GPUs that can be established based on the shared, on-chip interconnect channels. We first reverse-engineer the organization of the on-chip networks in modern GPUs to understand the core placements throughout the GPU. The hierarchical organization of the GPU results in the sharing of interconnect bandwidth between neighboring cores. Based on this understanding, we identify how contention for the interconnect bandwidth can be exploited for a novel covert channel attack. We propose two types of interconnect-based covert channels that exploit the on-chip network hierarchy. Unlike cache-based covert channels, no states of the on-chip network need to be modified for communication in our interconnect-based covert channel and the impact of contention is very predictable. By exploiting the parallelism of GPUs, our proposed covert channel results in very high bandwidth – achieving approximately 24 Mbps of bandwidth on NVIDIA Volta GPUs and results in one of the highest known microarchitectural covert channel bandwidth.

Memory placement in NUMA systems has a significant impact on overall performance; however, most prior work has not considered the performance impact of NUMA interconnect as a deterministic routing have been used to access a remote memory. In this work, we propose adaptive routing in NUMA interconnect to exploit path diversity in NUMA systems. In particular, we propose a software-based adaptive routing where packet routing paths are changed by modifying the routing table dynamically during runtime - thus, reducing interconnect channel contention and improving overall performance. Adaptive routing does not minimize memory contention but since the interconnect channel can be shared by multiple NUMA nodes and creates contention, adaptive routing maximizes the interconnect bandwidth by load-balancing traffic across different interconnect channels. We provide a proof-of-concept implementation of software-based adaptive routing on AMD Hypertransport-based system and demonstrate performance benefits, including how it is orthogonal to thread and memory schedulers and complements existing OS (or NUMA) schedulers.

NUMA (non-uniform memory access) servers are commonly used in high-performance computing and datacenters. Within each server, a processor-interconnect (e.g., Intel QPI, AMD HyperTransport) is used to communicate between the different sockets or nodes. In this work, we explore the impact of the processor-interconnect on overall performance -- in particular, the performance un- fairness caused by processor-interconnect arbitration. It is well known that locally-fair arbitration does not guarantee globally-fair bandwidth sharing as closer nodes receive more bandwidth in a multi-hop network. However, this work demonstrates that the opposite can occur in a commodity NUMA server where remote nodes receive higher bandwidth (and perform better). We analyze this problem and iden- tify that this occurs because of external concentration used in router micro-architectures for processor-interconnects without globally-aware arbitration. While accessing remote memory can occur in any NUMA system, performance un- fairness (or performance variation) is more critical in cloud computing and virtual machines with shared resources. We demonstrate how this unfairness creates significant performance variation when a workload is executed on the Xen virtualization platform. We then provide analysis using synthetic workloads to better understand the source of unfair- ness and eliminate the impact of other shared resources, including the shared last-level cache and main memory. To provide fairness, we propose a novel, history-based arbitration that tracks the history of arbitration grants made in the previous history window. A weighted arbitration is done based on the history to provide global fairness. Through simulations, we show our proposed history-based arbitration can provide global fairness and minimize the processor- interconnect performance unfairness at low cost.

Servers that consist of multiple nodes and sockets are interconnected together with a high-bandwidth, low latency processor interconnect network, such as Intel QPI or AMD Hypertransport technologies. The different nodes exchange packets through routers which communicate with other routers. A key component of a router is the routing table which determines which output port an arriving packet should be forwarded through. However, because of the flexibility (or programmability) of the routing tables, we show that it can result in security vulnerability. We describe the procedures for how the routing tables in a processor-interconnect router can be modified. Based on these modifications, we propose new system attacks in a server, which include both performance attacks by degrading the latency and/or the bandwidth of the processor interconnect as well as a livelock attack that hangs the system. We implement these system on an 8-node AMD server and show how performance can be significantly degraded. Based on this vulnerability, we propose alternative solutions that provide various trade-off in terms of flexibility and cost while minimizing the routing table security vulnerability.