Neural network (NN) accelerators with multi-chipmodule (MCM) architectures enable integration of massive computation capability; however, they face challenges of computing resource underutilization and off-chip communication overheads. Traditional parallelization schemes for NN inference on MCM architectures, such as intra-layer parallelism and inter-layer pipelining, show incompetency in breaking through both challenges, limiting the scalability of MCM architectures. We observed that existing works typically deploy layers separately rather than considering them jointly. This underexploited dimension leads to compromises between system computation and communication, thus hindering optimal utilization, especially as hardware/software scale. To address this limitation, we propose Scope, a merged pipeline framework incorporating this overlooked multi-layer dimension, thereby achieving improved throughput and scalability by relaxing tradeoffs between computation, communication and memory costs. This new dimension, however, adds to the complexity of design space exploration (DSE). To tackle this, we develop a series of search algorithms that achieves exponential-to-linear complexity reduction, while identifying solutions that rank in the top 0.05% of performance. Experiments show that Scope achieves up to $1.73 \times$ throughput improvement while maintaining similar energy consumption for ResNet-152 inference compared to state-of-the-art approaches.

The advancements in learning-based embedding models have propelled the embedding vector to become a vital data structure for various semantic search applications. The graph-based vector search method, which relies on the k-nearest neighbor (k-NN) graph, has gained immense popularity due to its superior performance in recall and throughput compared to other vector search methods. However, the construction of the k-NN graph is a time-consuming process, which not only affects the efficiency of identifying the optimal k-NN graph but also impacts data updates, especially for large-scale vector datasets. Despite the availability of various software-level optimization methods, such as iterative, refinement, and incremental methods, their peak performance is still limited to the underlying CPU platform. Furthermore, the existing GPU and specialized accelerator solutions only focus on partial k-NN graph construction methods and still encounter low efficiency when executing iterative methods due to irregular memory access, low data reuse, and redundant distance calculations.

Deep neural networks have shown outstanding performance on complex tasks. Recently, various researches have been developed to pursue fast and energy-efficient deep learning accelerators. However, devices may suffer from hard defects and hardware variability during its lifetime, which poses severe challenges to deep learning accelerators. To protect edge deep learning accelerators from fault-induced failures, we leverage the adversarial deep learning technique to tailor a lightweight online fault detection method for neural network accelerator chips. The proposed Adversarial Testing scheme (AT) is a function-level testing method outcompeting conventional test in several ways: negligible run-time overhead, super sensitivity to subtle hardware variations, which reduces chip over-kills and also the unnecessary diagnosis operations. The evaluation results show that AT can accurately detect fault occurrence and ensure the normal use of deep learning accelerator during its lifetime.

It has recently become trendy for neural network accelerators to scale from single-core to chiplet-based multichip architecture, as the growth of neural network depth and complexity are calling for the promotion of computation and memory capabilities. However, the unintended extensive inter-chip communication of chiplet-based accelerator may bottleneck the parallelism of deep learning inference, which is undesirable for many real-time applications and energy-efficient devices. Although it is imperative for novel schemes to be devised to alleviate this problem, related works are scarce. In this work, we present CampQ, a fine-grained communication-aware mixed-precision quantization method to accelerate inference parallelization by reducing the major inter-chiplet communication overhead. By leveraging the AutoML technique, CampQ is capable of determining different bit-width to activation groups according to thier transmission distances in on-package network. The experimental results show 1.4×-2.6× performance benefits and 29%-60% energy reduction over the 16-bit models for various neural networks and parallelism approaches.

Real-time inference of deep learning models on embedded and energy-efficient devices becomes increasingly desirable with the rapid growth of artificial intelligence on edge. Specifically, to achieve superb energy-efficiency and scalability, efficient parallelization of single-pass deep neural network (DNN) inference on chip multiprocessor (CMP) architectures is urgently required by many time-sensitive applications. However, as the number of processing cores scales up and the performance of cores has grown much fast, the on-chip inter-core data movement is prone to be a performance bottleneck for computation. To remedy this problem and further improve the performance of network inference, in this work, we introduce a communication-aware DNN parallelization technique called CAP, by exploiting the elasticity and noise-tolerance of deep learning algorithms on CMP. Moreover, in the hope that the conducted studies can provide new design values for real-time neural network inference on embedded chips, we also have evaluated the proposed approach on both multi-core Neural Network Accelerators (NNA) chips and general-purpose chip-multiprocessors. Our experimental results show that the proposed CAP can achieve 1.12×-1.65× system speedups and 1.14×-2.70× energy efficiency for different neural networks while maintaining the inference accuracy, compared to baseline approaches.

Accelerating deep neural networks on resource-constrained embedded devices is becoming increasingly important for real-time applications. However, in contrast to the intensive research works on specialized neural network inference architectures, there is a lack of study on the acceleration and parallelization of deep learning inference on embedded chip-multiprocessor architectures, which are favored by many real-time applications for superb energy-efficiency and scalability. In this work, we investigate the strategies of parallelizing single-pass deep neural network inference on embedded on-chip multi-core accelerators. These methods exploit the elasticity and noise-tolerance features of deep learning algorithms to circumvent the bottleneck of on-chip inter-core data moving and reduce the communication overhead aggravated as the core number scales up. The experimental results show that the communication-aware sparsified parallelization method improves the system performance by 1.6×−1.1× and achieves 4×−1.6× better interconnects energy efficiency for different neural networks.

The resurrection of Processing in memory (PIM) architectures is expected to address the ever-worsening memory wall issue in big data era. In this work, we propose XORiM, an inexpensive PIM design to achieve fast bulky bitwise XOR operation in commodity DRAM devices for memory-intensive workloads. Instead of resorting to 3D-integration or emerging memory technology, we reuse and adapt the peripheral circuits and row-buffers in memory to enable within-DRAM data manipulation. The implemented mechanism can also be employed to conduct high-throughput bulky data operations including memory initialization, AND, OR and INV. We present the detailed circuitry design and transistor-level simulation to evaluate the proposed method, and demonstrate the application of XO-RiM to realistic workloads by conducting full-system level simulation. The experimental results on data-intensive applications such as deduplication and data encryption show that about 1.5× and 5.1× overall performance benefits and 4.9× and 9.1× overall energy savings are achieved respectively by XORiM over conventional computing systems.