Sparse Deep Neural Networks (DNN) offer a large improvement in model storage requirements, execution latency and execution throughput. DNN pruning is contingent on knowing model weights, so networks can be pruned only after training. A priori sparse neural networks have been proposed as a way to extend sparsity benefits to the training process as well. Selecting a topology a priori is also beneficial for hardware accelerator specialization, lowering power, chip area, and latency.We present NeuroFabric, a hardware-ML model co-design approach that jointly optimizes a sparse neural network topology and a hardware accelerator configuration. NeuroFabric replaces dense DNN layers with cascades of sparse layers with a specific topology. We present an efficient and data-agnostic method for sparse network topology optimization, and show that parallel butterfly networks with skip connections achieve the best accuracy independent of sparsity or depth. We also present a multi-objective optimization framework that finds a Pareto frontier of hardware-ML model configurations over six objectives: accuracy, parameter count, throughput, latency, power, and hardware area.

I/O efficiency is crucial to productivity in scientific computing, but the growing complexity of HPC systems and applications complicates efforts to understand and optimize I/O behavior at scale. Data-driven machine learning-based I/O throughput models offer a solution: they can be used to identify bottlenecks, automate I/O tuning, or optimize job scheduling with minimal human intervention. Unfortunately, current state-of-the-art I/O models are not robust enough for production use and underperform after being deployed. We analyze four years of application, scheduler, and storage system logs on two leadership-class HPC platforms to understand why I/O models underperform in practice. We propose a taxonomy consisting of five categories of I/O modeling errors: poor application and system modeling, inadequate dataset coverage, I/O contention, and I/O noise. We develop litmus tests to quantify each category, allowing researchers to narrow down failure modes, enhance I/O throughput models, and improve future generations of HPC logging and analysis tools.

Emerging applications, like cloud services, are demanding more computational power, while also giving rise to various security and privacy challenges. Current multi-/many-core chip designs boost performance by using Networks-on-Chip (NoC) based architectures. Although NoC-based architectures significantly improve communication concurrency, they have thus far lack adequate security mechanisms such as enforceable process isolation. On the other hand, new security-aware architectures that protect applications and sensitive services in isolated execution environments, i.e., enclaves, have not been extended to provide comprehensive protection for NoC platforms. These enclave-based architectures (i) lack secure enclave-device interaction, (ii) cannot include unmodifiable third-party IP, or (iii) provide flexible enclave memory management.To address these design challenges, we introduce a new hardware security primitive, the Distributed Memory Guard, and design the first security architecture that protects sensitive services in NoC-based enclaves. We provide evaluation of this reference architecture and highlight the fact that one can design a scalable (i.e., NoC-based) and secure (i.e., enclave-based) architecture with minimal hardware complexity and system performance overhead.

With the growing complexity of high-performance computing (HPC) systems, achieving high performance can be difficult because of I/O bottlenecks. We analyze multiple years' worth of Darshan logs from the Argonne Leadership Computing Facility's Theta supercomputer in order to understand causes of poor I/O throughput. We present Gauge: a data-driven diagnostic tool for exploring the latent space of supercomputing job features, understanding behaviors of clusters of jobs, and interpreting I/O bottlenecks. We find groups of jobs that at first sight are highly heterogeneous but share certain behaviors, and analyze these groups instead of individual jobs, allowing us to reduce the workload of domain experts and automate I/O performance analysis. We conduct a case study where a system owner using Gauge was able to arrive at several clusters that do not conform to conventional I/O behaviors, as well as find several potential improvements, both on the application level and the system level.

Network-on-chips (NoCs) are vulnerable to transient and permanent faults caused by thermal violations, aging effects, component wear out, or even transient fault sources. Although some of these faults are addressed by previous research, we show that there are reliability threats in 3-D NoCs that go beyond the reliability issues investigated in 2-D interconnect networks. First, we highlight one such class of reliability threats and discuss their manifestations in 3-D NoCs. Second, we propose a thermal, reliability, and performance-aware routing algorithm to tackle: 1) previously established fault models and 2) the new highlighted class of reliability threats in partially connected 3-D NoCs. The proposed routing algorithm takes into account the states of routers and both the horizontal and through silicon via (TSV) links, along with the temperatures of routers and cores. It then routes the packets around failed or overheated links and routers, achieving lower latencies by avoiding misrouting. To achieve this, the proposed routing algorithm uses the concept of vertical link announcement to inform nodes in the network of the working condition of vertical links. We evaluate the proposed routing algorithm under a wide range of working conditions using the access Noxim NoC simulator. Results show that the proposed routing algorithm: 1) is able to tolerate almost any number and pattern of vertical link failures; 2) is reliable against the newly identified reliability threats; and 3) improves the latency and temperature distribution of the network compared to previously proposed routing algorithms.

The deployment of deep neural network (DNN) models is generally hindered by their training time. DNN training throughput is commonly limited by the fully-connected layers. This is due to their large size and low data reuse. Large batch sizes are often used to mitigate some of the effects. Increasing batch size can however hurt model accuracy, creating a tradeoff between accuracy and efficiency. We tackle the problem of training DNNs in on-chip memory, allowing us to train models without the use of batching. Pruning and quantizing dense layers can greatly reduce network size, allowing models to fit on the chip, but can only be applied after training. We propose a fully-connected but sparse layer that reduces the memory requirements of DNNs without sacrificing accuracy. We replace a dense matrix with a sparse matrix product with a predetermined topology. This allows us to: (1) train significantly smaller networks without a loss in accuracy, and (2) store weights without having to store connection indices. We therefore achieve significant training speedups due to the fast access to on-chip weights, smaller network size, and a reduced amount of computation per epoch.

In this paper, we introduce a neural network based predictive routing algorithm for on-chip networks which uses anticipated global network state and congestion information to efficiently route network traffic. The core of the algorithm is a multi-layer neural network machine learning approach where the inputs are level of occupancy of virtual channels, average latency for a particular router to be selected for route computation, the probability of virtual channel allocation, and the probability of winning switch arbitration at the crossbar. The algorithm lends itself to both node routing and source routing. To evaluate the PreNoc routing algorithm, we simulate both synthetic traffic and real application traces using a cycle-accurate simulator. In most test cases, the proposed approach outperforms current deterministic and adaptive routing techniques in terms of latency and throughput. The hardware overhead for supporting the new routing algorithm is minimal.

This work presents a cross-layer design of an adaptive manycore architecture to address the computational needs of emerging big data applications within the technological constraints of power and reliability. From the circuits end, we present links with reconfigurable repeaters that allow single-cycle traversals across multiple hops, creating fast single-cycle paths on demand. At the microarchitecture end, we present a router with bi-directional links, unified virtual channel (VC) structure, and the ability to perform self-monitoring and self-configuration around faults. We present our vision for self-aware manycore architectures and argue that machine learning techniques are very appropriate to efficiently control various configurable on-chip resources in order to realize this vision. We provide concrete learning algorithms for core and NoC reconfiguration; and dynamic power management to improve the performance, energy-efficiency, and reliability over static designs to meet the demands of big data computing. We also discuss future challenges to push the state-of-the-art on fully adaptive manycore architectures.