Zhirong Chen, Kaiyan Chang, Zhuolin Li, Cangyuan Li, Xinyang He, Chujie Chen, Mengdi Wang, Haobo Xu, Yinhe Han, Huawei Li, Ying Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large language model (LLM) inference demands vast memory capacity and hierarchical memory structures, but conventional architectures suffer from excessive energy and latency costs due to frequent data movement across deep memory tiers. To address this, we propose a wafer-scale SRAM-based Computing-in-Memory (CIM) architecture that performs all LLM operations in situ within the first-level SRAM, eliminating off-chip data migration and achieving unprecedented energy efficiency. However, wafer-scale SRAM CIM presents multiple challenges due to the limited first-level memory capacity, which requires efficient compute-memory resource allocation.

Fully Homomorphic Encryption (FHE) is a powerful privacy-preserving technology enabling secure computation on encrypted data, but it suffers from substantial performance overheads. Running FHE efficiently typically requires developing dedicated FHE accelerators, which can be costly and inflexible. Instead of pursuing entirely new accelerators, this paper explores an alternative paradigm: augmenting generic computing devices with a modular FHE-specific hardware extension unit (HEU) to create an efficient FHE engine. To make this paradigm viable, we propose three key innovations: (1) recognizing that some FHE operators are data-intensive and involve a massive volume of ciphertexts, we design the HEU with a 3D stacked memory-based architecture to handle data-intensive operators.We also provide software-level support to facilitate deploying FHE tasks on this extension-based architecture. (2) To capitalize on the hardware parallelism, we propose an adaptive offloading algorithm that intelligently distributes FHE operators between the computing device and the HEU. (3) To optimize the data layout and minimize the inter-tile data communications in the novel 3D stack memory, we propose a dedicated ciphertext mapping mechanism. Experimental results demonstrate that our work achieves substantial acceleration in FHE tasks.

Transformer-based models dominate modern AI workloads but exacerbate memory bottlenecks due to their quadratic attention complexity and ever-growing model sizes. Existing accelerators, such as Groq and Cerebras, mitigate off-chip traffic with large on-chip caches, while algorithmic innovations such as FlashAttention fuse operators to avoid materializing large attention matrices. However, as off-chip traffic decreases, our measurements show that on-chip SRAM accesses account for over 60% of energy in long-sequence workloads, making cache access the new bottleneck. We propose 3D-Flow, a hybrid-bonded, 3D-stacked spatial accelerator that enables register-to-register communication across vertically partitioned PE tiers. Unlike 2D multi-array architectures limited by NoC-based router-to-router transfers, 3D-Flow leverages sub-10 μm vertical TSVs to sustain cycle-level operator pipelining with minimal overhead. On top of this architecture, we design 3D-FlashAttention, a fine-grained scheduling method that balances latency across tiers, forming a bubble-free vertical dataflow without on-chip SRAM round-trips. Evaluations on Transformer workloads (OPT and QWEN models) show that our 3D spatial accelerator reduces 46–93% energy consumption and achieves 1.4×–7.6× speedups compared to state-of-the-art 2D and 3D designs.

Modern AI inference interleaves compute-dense matrix kernels with memory-sensitive element-wise and reduction operators. Symmetric RISC-V vector processors execute this mix with a uniform lane design, replicating heavy arithmetic units even when many instructions need only lightweight arithmetic. This paper presents AsymVPU, a hardware-software co-designed vector architecture that introduces fine-grained intra-lane asymmetry: one main processing element retains full FP64/FMA capability, while three auxiliary processing elements provide dense support for lightweight vector operations. AsymVPU couples this lane organization with a two-level hierarchical reduction network, a coordinated global load/store path, and compiler-inserted density hints that preserve the RVV programming abstraction. Implemented in SystemVerilog and evaluated in a same-flow 12 nm comparison against an Ara-derived symmetric baseline, AsymVPU achieves up to 2.4 × higher compute density for quantized workloads and 60% lower reduction latency, while retaining competitive performance on FMA-dominated kernels.

As ultra-large-scale integrated circuits continue to evolve, advanced packaging has become a promising approach to boost system performance, especially in high-performance computing applications. To support heterogeneous integration, multiple chiplets are integrated via large-area silicon interposers. However, due to reticle size limitations, interposer lithography requires multi-reticle stitching, which introduces high manufacturing costs and alignment-induced yield degradation. In this paper, we propose CIT-R3, a reticle-reuse-driven router that formulates routing and reticle reuse as a differentiable optimization problem. Additionally, a graph-patching algorithm is applied to enforce layout consistency across reused reticle regions. Experimental results on multi-chiplet benchmarks demonstrate significant reticle reuse improvements with minimal routing cost overhead.

Numerous quantum algorithms assume that classical data has already been converted into quantum states, a process known as Quantum State Preparation (QSP). However, achieving precise QSP requires a circuit depth that scales exponentially with the number of qubits, posing a significant challenge to realizing quantum advantage. Recent research explores Parameterized Quantum Circuits (PQCs) as an approximate alternative, offering improved scalability with reduced circuit depth. However, the iterative, state-by-state optimization required by this approach creates substantial runtime overhead, which severely limits its practicality.To improve the efficiency of approximate QSP, we introduce a novel two-stage framework that can potentially generate QSP circuits for arbitrary quantum states. In theoffline training stage, our model learns a direct mapping from target states to circuit parameters, thereby bypassing the need foronline, state-by-state optimizationduring theinference stage. Extensive evaluations show that our approach significantly reduces runtime overhead by up to 132×, making a steady step towards efficient neural approximate QSP.

Detecting malicious attacks is essential for protecting computer systems and ensuring device security. Federated Learning (FL)-based Intrusion Detection Systems (IDS) have emerged as promising solutions, enabling multiple clients (i.e., data owners) to collaboratively train intrusion detection models without sharing private data. However, current FL studies typically assume that each client’s training and test label distribution is identical. This assumption is overly idealistic and rarely holds in real-world scenarios, leading to suboptimal performance when label distribution shifts occur between the training and testing data. To address this challenge, we propose FedPRO, a plug-and-play framework designed to improve the test-time performance of existing FL methods, without modifying their original training pipelines or fine-tuning the trained FL models. Specifically, we develop a unique prototype generation and optimization mechanism to produce semantically meaningful class prototypes. These prototypes constitute a prototype memory bank, serving as an external knowledge repository. At test time, a prototype retrieval-augmented inference strategy is employed to query relevant prototypes and refine predictions on each client, effectively alleviating the label distribution shift issues and boosting prediction accuracy. We evaluate FedPRO by integrating it with various off-the-shelf FL methods on benchmark datasets. Extensive results consistently demonstrate its effectiveness in diverse settings. Notably, applying FedPRO to the state-of-the art method FedDBE improves its test accuracy from 79.25% to 86.66% on the CICIDS-2018 dataset, while introducing only approximately 32KB of additional communication overhead.

Fully Homomorphic Encryption (FHE) is a promising privacy-preserving technique that has drawn increasing attention from academia and industry. It allows computation directly on encrypted data without decryption. However, FHE incurs intensive computations. Chiplet-based designs integrate multiple processors, delivering high performance and thereby are embraced by computation-intensive FHE tasks. Despite the chiplet-based system with various processors, it is designed for unencrypted applications, falling short in handling FHE with unique ciphertext manipulations. One common approach to make it capable of FHE is developing a new FHE accelerator. However, this approach overlooks existing abundant resources already in the system and introduces a large area overhead. In this paper, we propose Chiplever, a framework that empowers a non-FHE-tailored system to efficiently support FHE tasks via a hardware extension. Chiplever aims to leverage the existing resources already in the room for FHE tasks. To achieve this, (1) Chiplever introduces a hardware extension with an FHE unit providing efficient function support for FHE operators. (2) Chiplever proposes an FHE coordinator in the extension, which enables direct ciphertext transfer between the newly introduced extension and existing chiplets, achieving efficient integration of the extension. (3) Chiplever lowers the high-level homomorphic operations to primitive operators that can be matched by existing chiplets and constructs a fine-grained computation graph. Based on this, Chiplever employs a task scheduling algorithm, which partitions the FHE task across the extension and existing chiplets to exploit the parallelism between them and reduce the ciphertext communication overheads. With these hardware and software optimizations, Chiplever achieves efficient FHE acceleration. Compared with prior FHE ASICs, Chiplever achieves 9.6× 15.9× speedup and 6.2× 67.4× throughput improvement on TFHE, while consuming only 18.8% 35.6% of the area overhead of dedicated FHE ASICs.

The decomposition-based chiplet design paradigm, which breaks monolithic system-on-chips (SoCs) into smaller chiplets, has proven effective in reducing costs and accelerating development. The vast design space of multi-chiplet systems necessitates automated techniques to maximize their benefits while minimizing overheads. However, current chiplet decomposition flows lack support for exploring reusable chiplets, missing opportunities to efficiently amortize non-recurring engineering costs. To fully unleash the economic potential of chiplets, we propose CHIPLETIZER 2.0, a cost-driven framework that guides SoC decomposition and enables chiplet reuse across a line of products. This early-stage chiplet planning framework determines critical system parameters involving the partition, floorplan, and D2D interface, based on user-specified optimization goals. Experimental results demonstrate that, compared with the existing chiplet decomposition techniques, our proposed framework achieves significant cost improvements on realistic product lines with acceptable overheads.

In post-Moore era, CMOS technology scaling has encountered enormous design and fabrication challenges. “Power Wall” limits the further increase of integration density. Emerging AI computing and data center deployments aggravate the power consumption problem further. In the pursuit of efficient computing paradigm, Nano Electro-mechanical (NEM) relay and Nanotube Random Access Memory (NRAM) technology have attracted enormous attention and have ultra-low power consumption compared to CMOS counterparts. NEM relay is a kind of device based on electronic and mechanical interaction switching, characterized by remarkably low power consumption. This article explores the application of NEM relay and NRAM technology to build a complex RISC processor, aiming to achieve much lower power without degrading performance. The controller and data path can be implemented with primitive logic gates made of NEM relays, and on-chip cache can be implemented with NRAM. Experimental results show that the energy efficiency of the processor design based on NEM relay and NRAM can be improved by 88.2% and 78.9% compared with CMOS technology based in-order and out-of-order microprocessors, respectively. Meanwhile, the performance can be improved by 42.9% and the instruction execution time can be reduced by more than 17.9%, which implies the potentials of NEM relay and NRAM for emerging ultra-low power applications.

This is a corrigendum for the article “A data-centric chip design agent framework for Verilog code generation” published in ACM Trans. Des. Autom. Electron Syst. 30, 6, Article 93 (October 2025), 27 pages.

With the slowdown of Moore’s Law, conventional monolithic chip architectures face challenges such as excessive die sizes and prohibitive manufacturing costs. Consequently, chiplets have emerged as a pivotal technology in the post-Moore era, attracting significant attention from both academia and industry. Multi-chiplet systems offer compelling advantages over monolithic ones, including enhanced integration density, reduced cost, and shortened time-to-market. However, realizing these benefits necessitates design flows capable of optimizing parameters across logical, physical, and circuit layers, which introduces substantial design complexity. Numerous design automation technologies have been proposed to address these challenges. This article provides a comprehensive overview of related advancements, categorizing chiplet design methodologies into two primary types: (i) top-down flows disintegrating existing hardware designs into chiplets and subsequently reintegrating them into multi-chiplet systems, and (ii) bottom-up flows combining existing chiplets into multi-chiplet systems based on user applications. This article begins by introducing foundational concepts, technical characteristics, and evaluation models relevant to multi-chiplet systems. We then systematically summarize the problem formulations, design spaces, and optimization techniques associated with top-down and bottom-up design flows. Finally, we discuss key challenges and potential future research directions in chiplet design automation, aimed at further harnessing the potential of chiplet-based integration.

The Number Theoretic Transform (NTT) significantly impacts the execution time of Fully Homomorphic Encryption (FHE) in practical applications, driving research into accelerated NTT methods. Computing-in-Memory (CIM) offers a promising solution to handle NTT's memory bottlenecks, yet efficiently implementing a CIM-based NTT engine remains challenging due to unique operations and large data sizes. We propose HP-CIM, a high-parallelism digital SRAM-based CIM NTT engine designed for large-scale NTT. HP-CIM integrates MVM-based NTT with a hierarchical SRAM architecture and novel even-odd data mapping, achieving nearly 3.08× faster execution and 4.96× energy savings compared to prior CIM-based designs.

Recent advances of large language models in the field of Verilog generation have raised several ethical and security concerns, such as code copyright protection and dissemination of malicious code. Researchers have employed watermarking techniques to identify codes generated by large language models. However, the existing watermarking works fail to protect RTL code copyright due to the significant syntactic and semantic differences between RTL code and software code in languages such as Python. This paper proposes a hardware watermarking framework RTLMarker that embeds watermarks into RTL code and deeper into the synthesized netlist. We propose a set of rule-based Verilog code transformations, ensuring the watermarked RTL code's syntactic and semantic correctness. In addition, we consider an inherent tradeoff between watermark transparency and watermark effectiveness and jointly optimize them. The results demonstrate RTLMarker's superiority over the baseline in RTL code watermarking.

Large language models (LLMs) are widely used for natural language understanding and text generation. An LLM model relies on a time-consuming step called LLM decoding to generate output tokens. Several prior works focus on improving the performance of LLM decoding using parallelism techniques, such as batching and speculative decoding. State-of-the-art LLM decoding has both compute-bound and memory-bound kernels. Some prior works statically identify and map these different kernels to a heterogeneous architecture consisting of both processing-in-memory (PIM) units and computation-centric accelerators (e.g., GPUs). We observe that characteristics of LLM decoding kernels (e.g., whether or not a kernel is memory-bound) can change dynamically due to parameter changes to meet user and/or system demands, making (1) static kernel mapping to PIM units and computation-centric accelerators suboptimal, and (2) one-size-fits-all approach of designing PIM units inefficient due to a large degree of heterogeneity even in memory-bound kernels.

Quantization is a widely-used compression technology to reduce the overhead of serving large language models (LLMs) on terminal devices and in cloud data centers. However, prevalent quantization methods, such as 8-bit weight-activation or 4-bit weight-only quantization, achieve limited performance improvements due to poor support for low-precision (e.g., 4-bit) activation. This work, for the first time, realizes practical W4A4KV4 serving for LLMs, fully utilizing the INT4 tensor cores on modern GPUs and reducing the memory bottleneck caused by the KV cache. Specifically, we propose a novel fine-grained mixed-precision quantization algorithm (FMPQ) that compresses most activations into 4-bit with negligible accuracy loss. To support mixed-precision matrix multiplication for W4A4 and W4A8, we develop a highly optimized W4Ax kernel. Our approach introduces a novel mixed-precision data layout to facilitate access and fast dequantization for activation and weight tensors, utilizing the GPU's software pipeline to hide the overhead of data loading and conversion. Additionally, we propose fine-grained streaming multiprocessor (SM) scheduling to achieve load balance across different SMs. We integrate the optimized W4Ax kernel into our inference framework, COMET, and provide efficient management to support popular LLMs such as LLaMA-3-70B. Extensive evaluations demonstrate that, when running LLaMA family models on a single A100-80G-SMX4, COMET achieves a kernel-level speedup of 2.88x over cuBLAS and a 2.02x throughput improvement compared to TensorRT-LLM from an end-to-end framework perspective.

Computing-in-memory (CIM) architectures demonstrate superior performance over traditional architectures. To unleash the potential of CIM accelerators, many compilation methods have been proposed, focusing on application scheduling optimization specific to CIM. However, existing compilation methods often overlook CIM's capability to switch dynamically between compute and memory modes, which is crucial for accommodating the diverse memory and computational needs of real-world deep neural network architectures, especially the emerging large language models. To fill this gap, we introduce CMSwitch, a novel compiler to optimize resource allocation for CIM accelerators with adaptive mode-switching capabilities, thereby enhancing the performance of DNN applications. Specifically, our approach integrates the compute-memory mode switch into the CIM compilation optimization space by introducing a new hardware abstraction attribute. Then, we propose a novel compilation optimization pass that identifies the optimal network segment and the corresponding mode resource allocations using dynamic programming and mixed-integer programming. CMSwitch uses the tailored meta-operator to express the compilation result in a generalized manner. Evaluation results demonstrate that CMSwitch achieves an average speedup of 1.31x compared to existing SOTA CIM compilation works, highlighting CMSwitch's effectiveness in fully exploiting the potential of CIM processors for a wide range of real-world DNN applications.

The billion-scale Large Language Models (LLMs) necessitate deployment on expensive server-grade GPUs with large-storage HBMs and abundant computation capability. As LLM-assisted services become popular, achieving cost-effective LLM inference on budget-friendly hardware becomes the current trend. This has sparked extensive research into relocating LLM parameters from expensive GPUs to external host memory. However, the restricted bandwidth between the host and GPU memory limits the inference performance of existing solutions. This work introduces Hermes, a budget-friendly system that leverages the near-data processing units (NDP) within commodity DRAM DIMMs to enhance the performance of a single consumer-grade GPU, achieving efficient LLM inference. We recognize that the inherent activation sparsity in LLMs naturally divides weight parameters into two categories, termed “hot” and “cold” neurons, respectively. Hot neurons, which consist of only approximately 20% of all weight parameters, account for 80% of the total computational load. In contrast, cold neurons make up the other 80% of parameters but are responsible for just 20% of the computational workload. Leveraging this observation, we propose a heterogeneous computing strategy: mapping hot neurons to a single computation-efficient GPU without large-capacity HBMs, while offloading cold neurons to NDP-DIMMs, which offer large memory size but limited computation capabilities. In addition, the dynamic nature of activation sparsity necessitates a real-time partition of hot and cold neurons and adaptive remapping of cold neurons across multiple NDP-DIMM modules. To tackle these issues, we introduce a lightweight predictor that ensures optimal real-time neuron partition and adjustment between GPU and NDP-DIMMs. Furthermore, we utilize a window-based online scheduling mechanism to maintain load balance among multiple NDP-DIMM modules. In summary, Hermes facilitates the deployment of LLaMA2-70B on consumer-grade hardware at a rate of 13.75 tokens/s and realizes an average 75.24 × speedup over the state-of-the-art offloading-based inference system on popular LLMs.

Large Language Models (LLMs) have emerged as the cornerstone of content generation applications due to their ability to capture relations between newly generated token and the full preceding context. However, this ability stems from the attention mechanism for decoding that retains the entire generation history as key value cache (KV cache). As the generated sequence lengthens, the KV cache expands, causing a substantial memory access bottleneck. In advanced LLM generation systems running on GPUs, the attention mechanism for decoding accounts for more than 50% of the total inference time when the KV cache length reaches 4096. To address this issue, this paper introduces LAD (Locality Aware Decoding), an LLM generation accelerator with algorithm-hardware enhancements that significantly decrease KV cache access, resulting in considerable speedups and energy savings. A key insight underlying LAD is that when the attention score for a specific position remains fixed over the next several decoding steps, it is unnecessary to repeatedly retrieve the associated key and value at each step to reproduce the computation. Our analysis reveals that numerous positions exhibit notable numerical locality in attention scores through multiple decoding steps. Leveraging these insights, we have designed an innovative attention decoding computation method that decreases the frequency of accessing the key and value for positions demonstrating good locality, all while maintaining decoding accuracy. Extensive experiments show that LAD generates sequences with an average ROUGE-1 similarity of 97% compared to those generated by the original model. When the length of KV cache exceeds 2048, the high configuration of LAD accelerator achieves on average (geomean) $10.7 \times$ speedup and $52.4 \times$ energy efficiency for the attention mechanism compared to the A100 GPU. For end-to-end model inference, it also achieves on average $2.3 \times$ speedup and $13.4 \times$ energy efficiency.

Neural Vector Search (NVS) has exhibited superior search quality over traditional key-based strategies for information retrieval tasks. An effective NVS architecture requires high recall, low latency, and high throughput to enhance user experience and cost-efficiency. However, implementing NVS on existing neural network accelerators and vector search accelerators is sub-optimal due to the separation between the embedding stage and vector search stage at both algorithm and architecture levels. Fortunately, we unveil that Product Quantization (PQ) opens up an opportunity to break separation. However, existing PQ algorithms and accelerators still focus on either the embedding stage or the vector search stage, rather than both simultaneously. Simply combining existing solutions still follows the beaten track of separation and suffers from insufficient parallelization, frequent data access conflicts, and the absence of scheduling, thus failing to reach optimal recall, latency, and throughput. To this end, we propose a unified and efficient NVS accelerator dubbed NeuVSA based on algorithm and architecture co-design philosophy. Specifically, on the algorithm level, we propose a learned PQ-based unified NVS algorithm that consolidates two separate stages into the same computing and memory access paradigm. It integrates an end-to-end joint training strategy to learn the optimal codebook and index for enhanced recall and reduced PQ complexity, thus achieving smoother acceleration. On the architecture level, we customize a homogeneous NVS accelerator based on the unified NVS algorithm. Each sub-accelerator is optimized to exploit all parallelism exposed by unified NVS, incorporating a structured index assignment strategy and an elastic on-chip buffer to alleviate buffer conflicts for reduced latency. All sub-accelerators are coordinated using a hardware-aware scheduling strategy for boosted throughput. Experimental results show that the joint training strategy improves recall by 4.6% over the separated strategy and accuracy by 43.5% over LUT-NN. NeuVSA achieves $2.82 \times$ to $416.17 \times$ lower latency over CPU, GPU, DFX+ANNA, and PQA+ANNA, and up to $49.60 \times$ and $10.57 \times$ higher average throughput over CPU and GPU, respectively. NeuVSA also reduces chip area by 65.2% over PQA+ANNA.

Layout Synthesis for Quantum Computing (LSQC) is a critical component of quantum design tools. Traditional LSQC studies primarily focus on optimizing for reduced circuit depth by adopting a device-centric design methodology. However, these approaches overlook the impact of classical processing and communication time, thereby being insufficient for Dynamic Quantum Circuits (DQC).To address this, we introduce CLASS, a controller-centric layout synthesizer designed to reduce inter-controller communication latency in a distributed control system. It consists of a two-stage framework featuring a hypergraph-based modeling and a heuristic-based graph partitioning algorithm. Evaluations demonstrate that CLASS effectively reduces communication latency by up to 100% with only a 2.10% average increase in the number of additional operations.

With the rise of fully homomorphic encryption-based private inference, data centers are anticipated to simultaneously handle two disparate computational demands: plaintext-based non-private inference (NPI) and ciphertext-based private inference (PI). Unfortunately, current solutions face challenges in addressing this trend. They either depend on costly, inflexible dedicated accelerators or utilize general-purpose hardware with inferior performance. This limitation underscores the urgent need for a unified architecture capable of serving both normal and privacy-sensitive users with high efficiency.However, the fundamental disparities in computation patterns and resource management between NPI and PI make their architectural fusion intricate. To bridge this gap, we explore their inherent similarities and apply fine-grained reconfiguration to maximize resource sharing. We propose RTPU, a reconfigurable multi-core architecture that can seamlessly switch between tensor-based plaintext and polynomial ring-based ciphertext computations. Building upon its reconfigurable computing fabric and parallelization mechanism, we introduce a kernel group-based scheduling strategy to optimize hardware utilization and QoS. Experimental results show that: i) The RTPU architecture achieves near-ASIC performance and beyond-ASIC flexibility with substantial silicon reuse between NPI and PI. ii) The RTPU scheduler sustains high resource utilization for multi-tenant workloads with varying privacy requirements.

The emergence of multimodal large language models (MLLMs) presents promising opportunities for automation and enhancement in Electronic Design Automation (EDA). However, comprehensively evaluating these models in circuit design remains challenging due to the narrow scope of existing benchmarks. To bridge this gap, we introduce MMCircuitEval, the first multimodal benchmark specifically designed to assess MLLM performance comprehensively across diverse EDA tasks. MMCircuitEval comprises 3614 meticulously curated question-answer (QA) pairs spanning digital and analog circuits across critical EDA stages—ranging from general knowledge and specifications to front-end and back-end design. Derived from textbooks, technical question banks, datasheets, and real-world documentation, each QA pair undergoes rigorous expert review for accuracy and relevance. Our benchmark uniquely categorizes questions by design stage, circuit type, tested abilities (knowledge, comprehension, reasoning, computation), and difficulty level, enabling detailed analysis of model capabilities and limitations. Extensive evaluations reveal significant performance gaps among existing LLMs, particularly in back-end design and complex computations, highlighting the critical need for targeted training datasets and modeling approaches. MMCircuitEval provides a foundational resource for advancing MLLMs in EDA, facilitating their integration into real-world circuit design workflows. Our benchmark is available at https://github.com/cure-lab/MMCircuitEval.

The emergence of chiplet technology has elevated the IO chiplet to a critical component for scalable system I/O and memory access extension. There is a growing demand for more effective and flexible memory management strategies enabled by the IO chiplet, to support the increasing scale and complexity of chiplet-based systems. Currently, many clustering-based memory access architectures that rely on static or coarse-grained memory clustering struggle with imbalanced memory workloads and cross-chiplet communication, which degrade performance due to resource contention and increased latency. To address this challenge, we propose FlexIO, a scalable IO chiplet architecture with flexible memory controller mapping, optimized for scalable chiplet-based systems with non-uniform memory access. FlexIO introduces a hardware-based application-to-memory controller mapping layer, which enables reconfigurable mapping of memory requests for each application to specified memory controllers. This flexibility is further enhanced by a heuristic-based optimization framework that explores optimal memory mappings tailored to the target chiplet-based system. Experiments demonstrate that FlexIO with the optimization framework improves 24.6 % system throughput, reduces 30.1 % average memory access latency, and decreases$\mathbf{1 6. 5 \%}$worst-case memory access latency compared to prior designs.

One-shot post-training pruning enhances the deployment of billion-scale large language models (LLMs), with the pruning metric playing a pivotal role in determining which weights to remove. However, existing metrics underperform due to their reliance on a simple symbolic combination of weights and activations, overlooking imbalanced weight magnitudes and the disproportionate influence of activation outliers. To overcome these limitations, we introduce \textbf{BaWA}, a novel pruning metric that systematically \underline{Ba}lances \underline{W}eight and \underline{A}ctivation distributions for more effective pruning. BaWA introduces two key innovations: \textbf{magnitude normalization}, which mitigates weight imbalance across channels for fairer pruning decisions, and \textbf{outlier regularization}, which reduces the impact of activation outliers, ensuring more appropriate channel prioritization. To further enhance its effectiveness, BaWA incorporates an efficient and automatic framework for optimizing normalization and regularization hyperparameters. Extensive experiments validate BaWA as a state-of-the-art (SOTA) pruning metric. For instance, applying BaWA to induce 2:4 sparsity in Mistral-7B reduces perplexity in language comprehension by 2.49 and improves average downstream task accuracy by 3.08\%, outperforming the previous SOTA method Wanda.

Ensuring data integrity is critical for cloud-native databases (CNDs), which typically adopt a disaggregated architecture separating compute and storage. These systems rely on distributed file systems (DFS) for persistence, where the storage backend plays a key role in overall performance and reliability. However, existing checksum mechanisms either protect only local I/O scopes or impose significant overhead when applied end-to-end. In this paper, we present PStore, a high-performance, integrity-aware storage backend module that provides end-to-end protection through an optimized checksum architecture. By leveraging the structured I/O patterns of CNDs and employing deferred checksum recalculation, PStore significantly reduces both I/O and computational overhead. We implement PStore and evaluate it on real hardware. Results show that PStore improves performance by up to 168.2 % over existing integrity-aware solutions, while fault injection experiments confirm its robust error detection, demonstrating that strong integrity guarantees can coexist with high performance.

Out-of-core graph processing systems are severely bottlenecked by I/O to the external storage because of the low compute-to-I/O ratio and the substantial amount of irregular data accesses. In order to alleviate the I/O bottleneck, prior works either focus on improving the bandwidth utilization by converting random I/O requests into sequential ones, or improving the data utilization by fetching only the required data to avoid the I/O redundancy. However, the former usually loads massive unused data, while the latter can induce frequent finegrained I/O requests, wasting the parallelism of the I/O channels and leading to under-utilization of the limited I/O bandwidth. Different from prior works, we systematically explore the use of computational storage devices (CSDs), which offer in-storage computing facilities with higher I/O bandwidth, to improve both the bandwidth utilization and data utilization for higher I/O efficiency. Specifically, we first introduce a graph-semanticaware data organization to enable the loading of only active graph partitions at the granularity of a physical page, reducing redundant I/O and enhancing data utilization. Additionally, we propose to coalesce parallel I/O requests of graph partitions distributed across different flash dies to maximize the parallelism of internal I/O channels, thereby fully utilizing the internal I/O bandwidth of CSDs. In addition, we capture the dynamic status of graph processing tasks across the iterations and partitions at runtime to dynamically offload I/O-intensive workloads into the instorage processors with restricted computing resources but higher I/O bandwidth to further improve the I/O efficiency. With the above techniques, we implement an out-of-core graph processing system prototype, namely TaijiGraph, on an open-channel CSD. According to our experiments on a set of representative graph datasets and algorithms, TaijiGraph achieves average speedups of$2.43 \times, 3.81 \times, 2.21 \times$and$7.89 \times$, respectively, when compared to state-of-the-art out-of-core graph processing systems including GridGraph, LUMOS, Blaze, and GraphSSD.

As the number of chiplets in 2.5D IC continues to increase, existing chiplet placement method faces two main challenges: (1) the combinatorial explosion in the search space, and (2) the difficulty of achieving global optimization through iterativing chiplet and terminal placement. To tackle these challenges, we develop an efficient analytical RDL chiplet-terminal co-placement algorithm, to ensure simultaneous placement of chiplets and terminals. Our algorithm employs RDL chiplet-terminal co-placement in three stages: analytical global placement, legalization, and bump-terminal assignment, to achieve high-quality placement results that comply with design rules. Experimental results demonstrate that our algorithm reduces average wirelength by 31% compared to prior work for the common testcases, with a maximum speedup of up to 6500× in testcases with more than 10 chiplets.

The design of a scalable Quantum Control Architecture (QCA) faces two primary challenges.First, the continuous growth in qubit counts has rendered distributed QCA inevitable, yet the nondeterministic latencies inherent in feedback loops demand cycleaccurate synchronization across multiple controllers.Existing synchronization strategies -whether lock-step or demand-drivenintroduce significant performance penalties.Second, existing quantum instruction set architectures are polarized, being either too abstract or too granular.This lack of a unifying design necessitates recurrent hardware customization for each new control requirement, which limits the system's reconfigurability and impedes the path toward a scalable and unified digital microarchitecture.Addressing these challenges, we propose Distributed-HISQ, featuring: (i) HISQ, A universal instruction set that redefines quantum control with a hardware-agnostic design.By decoupling from quantum operation semantics, HISQ provides a unified language for control sequences, enabling a single microarchitecture to support various control methods and enhancing system reconfigurability.(ii) BISP, a booking-based synchronization protocol that can potentially achieve zero-cycle synchronization overhead.The feasibility and adaptability of Distributed-HISQ are validated through its implementation on a commercial quantum control system targeting superconducting qubits.We performed a comprehensive evaluation using a customized quantum software stack.Our results show that BISP effectively synchronizes multiple control boards, leading to a 22.8% reduction in average program execution time and a ∼ 5× reduction in infidelity when compared to an existing lock-step synchronization scheme.

Precise and rapid performance prediction for dataflow-based accelerators is essential for efficient hardware design and design space exploration.However, existing methods often fall short due to limited generalization across hardware architectures, applications, and input-dependent control flows.Considering the rich program semantic knowledge contained in pre-trained large language models (LLMs), which is used for text and code generation, we propose a progressive numeric modeling paradigm based on pre-trained LLMs.This is an approach to achieve hardware, application, and control flow-sensitive generalization in dataflow accelerator performance prediction.Specifically, to make accurate performance estimates for unseen applications beyond the scope of the training data, we propose a numeric prediction model capable of estimating any performance range.This is achieved by treating the numerical data of the dataflow program as separate tokens and using categorical output for performance values, allowing us to observe confidence at each numerical position.Second, LLMulator supports input-adaptive performance prediction by introducing a reinforcement learning-based dynamic calibration framework, enabling accurate modeling of applications whose control flow varies with input-unlike prior methods that * Corresponding author.

Approximate nearest neighbor search (ANNS) is essential for applications like recommendation systems and retrieval-augmented generation (RAG) but is highly I/O-intensive and memory-demanding. CPUs face I/O bottlenecks, while GPUs are constrained by limited memory. DRAM-based Processing-in-Memory (DRAM-PIM) offers a promising alternative by providing high bandwidth, large memory capacity, and near-data computation. This work introduces DRIM-ANN, the first optimized ANNS engine leveraging UPMEM’s DRAM-PIM. While UPMEM scales memory bandwidth and capacity, it suffers from low computing power because of the limited processor embedded in each DRAM bank. To address this, we systematically optimize ANNS approximation configurations and replace expensive squaring operations with lookup tables to align the computing requirements with UPMEM’s architecture. Additionally, we propose load-balancing and I/O optimization strategies to maximize parallel processing efficiency. Experimental results show that DRIM-ANN achieves a 2.46× speedup over a 32-thread CPU and up to 2.67× over a GPU when deployed on computationally enhanced PIM platforms.

Due to the demonstrated superiority, dynamic neural networks (NNs), which adapt their network structures to different inputs, have been recognized as an optimized alternative to conventional static NNs. However, researchers have not explored the implications of dynamic NN on neural processing unit (NPU) architecture design. Consequently, we analyze the characteristics and inefficient sources of executing dynamic NNs on existing hardware. From our analysis, existing NPUs, designed for static NNs, cannot effectively handle the execution of dynamic operator and agent-dependent data loading in dynamic NNs.To this end, we present DNA, an efficient accelerator optimized to deal with the challenges of running general dynamic NNs. Firstly, to improve the execution efficiency of dynamic operators, we propose a transverter-based online scheduling strategy to rapidly generate efficient scheduling for each dynamic operator. Secondly, to mitigate hardware idleness caused by the non-deterministic and agent-dependent data access patterns in dynamic NNs, we propose a novel predictor-based prefetching strategy that achieves effective data preloading with negligible cost. We implemented our accelerator, DNA, by integrating an additional online scheduler into a typical many-core baseline accelerator. According to our evaluation of various dynamic NNs, DNA achieves 3.48× speedup and 3.03× energy savings over the baseline accelerator.

Hardware-aware Neural Architecture Search (HW-NAS) has garnered significant research interest due to its ability to automate the design of neural networks for various hardware platforms. Prevalent HW-NAS frameworks often use fast predictors to estimate network performance, bypassing the time-consuming actual profiling step. However, the resource-intensive nature of building these predictors and their accuracy limitations hinder their practical use in diverse deployment scenarios. In response, we emphasize the indispensable role of actual profiling in HW-NAS and explore efficiency optimization possibilities within the HW-NAS framework. We provide a systematic analysis of profiling overhead in HW-NAS and identify many redundant and unnecessary operations during the search phase. We then optimize the workflow and present Insitu NAS, which leverages similarity features and exploration history to eliminate redundancy and improve runtime efficiency. In-situ NAS also offers simplified interfaces to ease the user’s effort in managing the complex device-dependent profiling flow, enabling plug-and-search functionality across diverse hardware platforms. Experimental results show that In-situ NAS achieves an average 10x speedup across different hardware platforms while reducing the search overhead by 8x compared to predictor-based approaches in various deployment scenarios. Additionally, In-situ NAS consistently discovers networks with better accuracy (about 1.5%) across diverse hardware platforms compared to predictor-based NAS.

Resistive random access memory (RRAM) emerges as a promising technology for developing energy-efficient deep neural network (DNN) accelerators, owing to its analog computing paradigm for matrix-vector multiplication. However, the inherent nonideal device features of RRAM cells, such as device variation, read disturbances, and limited on/off ratio, present challenges for model deployment. Therefore, to ensure accurate storage and computing precision for RRAM-based accelerators, a widely used practice is encoding a DNN weight by multiple cells, resulting in significant memory overhead and underutilization. This challenge is further exacerbated by the rapid increases in model size witnessed in recent years. While the one-to-one weight-cell mapping strategy can improve memory utilization, it inevitably introduces deviations in the mapped DNN weight from the desired value due to RRAM variation issues, leading to model accuracy degradation. In response to this challenge, we abstract the model optimization on RRAM chips as a non-ideal PIM device optimization problem, aimed at optimizing model accuracy without the requirement of precise weight programming. We systematically analyze the model optimization behavior on multilevel RRAM devices by investigating the accuracy recovery process of various fine-tuning strategies in recovering model performance under the non-ideal PIM device setting. Based on the analysis, we propose a non-ideal PIM device finetune scheme to recover the model performance for multilevel RRAM under the non-ideal PIM device setting. Our proposed scheme leverages knowledge distillation and exploits input/output information of the model on RRAM to guide the fine-tuning process, finally restoring its accuracy. Experimental results demonstrate the efficacy of our non-ideal PIM device fine-tuning scheme, achieving nearly complete recovery of model performance. Our approach yields over a 3% improvement in model accuracy compared to variation-aware training approaches.

Due to the disorder of points, point clouds need to be structured by sampling and neighbor query before feeding to Deep Neural Networks (DNNs). Structuring point clouds costs high computation overhead, which limits the deployment of DNNs on embedded devices such as autonomous vehicles and robots. To address this problem, we design a novel data structure, i.e., Fast Spatial-Searching Tree (FSSTree), to accelerate point cloud structuring for DNNs on embedded devices. The FSSTree is constructed based on density distribution of point clouds to achieve semantic segmentation, which can guarantee that points with similar spatial positions are stored in adjacent storage sets. Based on FSSTree, we propose a point-sparsity-aware sampling method and a leafwise k-nearest neighbor query method to reduce the computation overhead of structuring point clouds. Meanwhile, the point-sparsity-aware sampling method achieves fair sampling on both dense and sparse parts, which can overcome the nonuniform distribution of point clouds caused by occlusion, lighting and other factors. The leafwise k-nearest neighbor query method skips a large number of dissimilar points to quickly obtain the neighbor points, which can significantly reduce the search scope. We also present a layerwise self-pruning algorithm to automatically adjust the FSSTree after each layer’s operation to match the hierarchical architecture of DNNs. Finally, we conduct extensive experiments on KITTI, S3DIS and ModelNet40 datasets and three devices (including an RTX 3090 server, a Jetson AGX Xavier and an Apple M2). The experimental results demonstrate the efficiency of our approach, which can reduce the time overhead by up to 97.46% compared with the other five methods. The code is released athttps://github.com/EmbeddedAILab-UESTC/fsstree.

Recent advances in large language models (LLMs) have demonstrated significant potential for automated hardware description language (HDL) code generation from high-level specifications. However, two critical challenges limit further progress in this domain: the scarcity of quality Verilog training data and the inability of current approaches to generate RTL code optimized for power, performance, and area (PPA) metrics. This article presents a comprehensive data-centric framework that addresses these limitations through innovations in both pre-fine-tuning data preparation and after-fine-tuning optimization strategies. In the pre-fine-tuning phase, we tackle the data scarcity problem with an automated design-data augmentation framework that generates high-volume, high-quality natural language specifications aligned with corresponding Verilog code and EDA scripts. Our approach creates a complete RTL-level feedback loop by augmenting EDA scripts, RTL code, and EDA tool feedback. In the after-fine-tuning phase, we focus on generating PPA-aware RTL code through a novel search and prompt framework. Our approach implements iterative filtering and selection of LLM-generated Verilog variants while providing high-quality predefined prompts, including composition and interface specifications. To evaluate the effectiveness of our data augmentation method, we fine-tune Llama 2-13B and Llama 2-7B models using the dataset generated by our augmentation framework. The results demonstrate a significant improvement in the Verilog generation tasks with LLMs. Moreover, the accuracy of Verilog generation surpasses that of the current state-of-the-art open-source Verilog generation model, increasing from 58.8% to 70.6% with the same benchmark. Our 13B model has a pass rate improvement compared with GPT-3.5 in Verilog generation and outperforms in EDA script (i.e., SiliconCompiler) generation with only 200 EDA script data. Additionally, to evaluate the effectiveness of the our agent framework, we compare the PPA on the GPT-3.5, where the results show that the agent refined RTL code can have a better quality than the generated RTL code only with GPT-3.5.

Hardware description language (HDL) code designing is a critical component of the chip design process, requiring substantial engineering and time resources. Recent advancements in large language models (LLMs), such as GPT series, have shown promise in automating HDL code generation. However, current LLM-based approaches face significant challenges in meeting real-world hardware design requirements, particularly in handling complex designs and ensuring code correctness. Our evaluations reveal that the functional correctness rate of LLM-generated HDL code significantly decreases as design complexity increases. In this article, we propose the AutoSilicon framework, which aims to scale up the hardware design capability of LLMs. AutoSilicon incorporates an agent system, which (1) allows for the decomposition of large-scale, complex code design tasks into smaller, simpler tasks; (2) provides a compilation and simulation environment that enables LLMs to compile and test each piece of code it generates; and (3) introduces a series of optimization strategies. Experimental results demonstrate that AutoSilicon can scale hardware designs to projects with code equivalent to over 10,000 tokens. In terms of design quality, it further improves the syntax correctness rate and functional correctness rate compared with approaches that do not employ any extensions. For example, compared to directly generating HDL code using GPT-4-turbo, AutoSilicon enhances the syntax correctness rate by an average of 35.8% and improves functional correctness by an average of 35.6%.

Graph-based deep learning point cloud processing has gained increasing popularity but its performance is dragged by the dominating graph construction (GC) phase with irregular computation and memory access. Existing works that accelerate GC by tailoring architecture for a single GC algorithm fail to maintain efficiency because they neglected the best GC algorithm variation incurred by the point-cloud density variation in changing scenarios. Therefore, we propose APoX, a unified architecture with an adaptive GC scheme that can identify the optimum GC approach according to the point cloud variation. Experiments indicate that APoX achieves higher performance and energy efficiency over existing accelerators.

The stagnation of Moore’s law stimulates the concept of breaking monolithic chips into smaller chiplets. However, tactic design partitioning remains an unaddressed issue despite its crucial role in chip product cost reduction. In this paper, we propose Chipletizer, a framework to guide the design partitioning for those who would benefit from chiplet reuse across a line of SoC products. The proposed generic framework supports the repartitioning of multiple SoCs into reusable chiplets economically and efficiently with user-specified parameters. Experimental results show that, compared with existing partitioning strategies, our proposed framework achieves notable cost improvement on realistic products with acceptable power and latency overheads.

In recent years, various computing-in-memory (CIM) processors have been presented, showing superior performance over traditional architectures. To unleash the potential of various CIM architectures, such as device precision, crossbar size, and crossbar number, it is necessary to develop compilation tools that are fully aware of the CIM architectural details and implementation diversity. However, due to the lack of architectural support in current popular open-source compiling stacks such as TVM, existing CIM designs either manually deploy networks or build their own compilers, which is time-consuming and labor-intensive. Although some works expose the specific CIM device programming interfaces to compilers, they are often bound to a fixed CIM architecture, lacking the flexibility to support the CIM architectures with different computing granularity. On the other hand, existing compilation works usually consider the scheduling of limited operation types (such as crossbar-bound matrix-vector multiplication). Unlike conventional processors, CIM accelerators are featured by their diverse architecture, circuit, and device, which cannot be simply abstracted by a single level if we seek to fully explore the advantages brought by CIM.

With the rapid up-scaling of transformer-based large language models (LLM), training these models is becoming increasingly demanding on novel parallel training techniques. Tensor partitioning is an extensively researched parallel technique, encompassing data and model parallelism, and has a significant influence on LLM training performance. However, existing state-of-the-art parallel training systems are based on incomplete tensor partitioning space, where the distribution of partitioned sub-operators is limited to the spatial dimension. We discover that introducing the temporal dimension into tensor partitioning of LLM training instance provides extra opportunities to avoid collective communication across devices, saving memory space and also overlapping device-to-device communication with computation. In this paper, we propose a new tensor partition primitive that distributes sub-operators along both the spatial and temporal dimensions to further explore communication and memory overhead reduction over current solutions. This new primitive creates a broader parallelization space and leads to parallel solutions that achieve better training throughput with lower peak memory occupancy compared to state-of-the-art techniques. To efficiently deploy optimized parallel transformer model training to multiple devices, we further present an optimization algorithm that can find optimal parallel solutions from our spatial-temporal tensor partition space with acceptable search time. Our evaluation shows that our optimized tensor partitioning achieves up to 1.68 × training throughput with 69% peak memory occupancy compared to state-of-the-art distributed training systems when training LLMs. Upon scaling to 32 GPUs, the geo-mean speedup across benchmarks is 1.30 ×. When applied in 3D parallelism, up to 1.46 × training throughput can be achieved.

Recent advances in large language models have demonstrated their potential for automated generation of hardware description language (HDL) code from high-level prompts. Researchers have utilized fine-tuning to enhance the ability of these large language models (LLMs) in the field of Chip Design. However, the lack of Verilog data hinders further improvement in the quality of Verilog generation by LLMs. Additionally, the absence of a Verilog and electronic design automation (EDA) script data augmentation framework significantly increases the time required to prepare the training dataset for LLM trainers. This paper proposes an automated design-data augmentation framework, which generates high-volume and high-quality natural language aligned with Verilog and EDA scripts. For Verilog generation, it translates Verilog files to an abstract syntax tree and then maps nodes to natural language with a predefined template. For Verilog repair, it uses predefined rules to generate the wrong verilog file and then pairs EDA Tool feedback with the right and wrong verilog file. For EDA Script generation, it uses existing LLM(GPT-3.5) to obtain the description of the Script. To evaluate the effectiveness of our data augmentation method, we finetune Llama2--13B and Llama2-7B models using the dataset generated by our augmentation framework. The results demonstrate a significant improvement in the Verilog generation tasks with LLMs. Moreover, the accuracy of Verilog generation surpasses that of the current state-of-the-art open-source Verilog generation model, increasing from 58.8% to 70.6% with the same benchmark. Our 13B model (ChipGPT-FT1) has a pass rate improvement compared with GPT-3.5 in Verilog generation and outperforms in EDA script (i.e., SiliconCompiler) generation with only 200 EDA script data.

Fully Homomorphic Encryption (FHE) is one of the most promising privacy-preserving techniques that has drawn increasing attention from academia and industry due to its ideal security. Chiplet-based designs integrate multiple dies into the package delivering high performance and thereby are embraced by the resources-hungry FHE. Despite the chiplet-based system with various specialized accelerators, it falls short in supporting FHE with the novel polynomial operations. For a chiplet-based system that is not tailored for FHE, one common approach to support FHE is designing a new dedicated accelerator, However, this full design-and-build approach overlooks the existing abundant resources of accelerators in the system and incurs repeated customization and resource waste.

Quantization is one of the most hardware-efficient ways to reduce inference costs for deep neural network (DNN) models. Nevertheless, with the continuous increase of DNN model sizes (240× in two years) and the emergence of large language models, existing static quantization methods fail to utilize the sparsity and redundancy of models sufficiently. Motivated by the pervasive dynamism in data tensors across DNN models, we propose a dynamic precision quantization algorithm to further reduce computational costs beyond statically quantized DNN models. Furthermore, we find that existing precision-flexible accelerators cannot support the DNN models with dynamic precision. To this end, we design a novel accelerator, Drift, and achieve online scheduling to efficiently support dynamic precision execution. We conduct experiments with various DNN models, including CNN-based and Transformer-based models. Evaluation results show that Drift achieves 2.85× speedup and 3.12× energy saving compared to existing precision-flexible accelerators with statically quantized models.

Graph convolutional network (GCN) has been proven powerful in various tasks for it combines both neural networks and graph processing operators. However, this characteristic makes GCN exhibit hybrid execution patterns, which is unfavorable for CPUs and GPUs. Therefore, designing specialized GCN accelerators is becoming a prevalent paradigm. Unfortunately, as graph scale continues to grow, existing GCN accelerators suffer from significant bandwidth consumption and memory footprint as they neglect the inherent semantic redundancy of vertex features. Although applying Product Quantization to GCN is a promising solution to reduce the sizeable graph data via distilling semantic redundancy, it introduces novel operations with unique patterns that existing GCN accelerators cannot support. In this paper, we propose GPACE, an energy-efficient GCN accelerator that can fully harness the potential of PQ to reduce bandwidth consumption and data movement. GPACE is designed with a lookup-efficient architecture and well-optimized dataflow to support the unique data access and computation pattern of PQ-GCN. In addition to leveraging PQ to distill semantic redundancy, we exploit the operation redundancy and propose a redundancy-aware architecture to detect and reduce types of redundant operations to achieve higher energy efficiency. Evaluations show GPACE achieves high speedup and energy saving compared with CPU, GPU, and specialized GCN accelerators.

ReRAM-based accelerator of bit-slicing architecture is a promising solution to neural network inference, which allows ineffectual bit-operation removal for greater potential gains. However, existing techniques mostly exploit the removal of weight-associated ineffectual operations, which cannot eliminate the activation-induced ineffectual operations. Alternatively, some approaches adopt an isolated two-stage approach to remove at the weight and activation-level, which leaves a big proportion of ineffectual bit-level operations. Therefore, in contrast to all these coarse-grained operation removal techniques, it is challenging to jointly eliminate ineffectual bit-operation induced by either activation or weight bit-slices for ReRAM-based accelerators. This work presents a novel ineffectual bit-operation removal approach and the accompanied ReRAM-based bit-operation clipping architecture that skips all those bit-level operations that make negligible impacts on neural network outputs. In experiments, the proposed bit-operation clipping ReRAM accelerator, Bit-Trimmer, achieves$5.28 \times$energy efficiency and$2.04 \times$speedup on average. Besides, compared with two SOTA ReRAM accelerator designs with bit-operation removal, it outperforms by$1.56 \times$and$1.88 \times$energy efficiency.

An Large Language Model (LLM)-based question-answering (QA) system has gained attention for its conversational ability. However, domain knowledge limitations, time lag, high training costs, and security concerns suggest building on-premise QA systems with embedding techniques. However, deploying embedding-based QA systems on existing GPUs or domain-specific accelerators is sub-optimal as they only address high computation costs and ignore large memory footprint and data movement costs, which impact response latency and user experience. To address these issues, we propose a hybrid near-data processing platform, HyQA, which collaboratively optimizes response latency, memory footprint, and data movement cost by exploiting the benefit of near-memory and near-storage computing simultaneously. First, HyQA analyzes computational patterns of sub-tasks in embedding-based QA systems, tailors domain-specific hardware accelerators, and assigns suitable computational paradigms. Second, these dedicated accelerators are designed to communicate directly with flash memory, avoiding additional data movement. The experiment shows that HyQA significantly improves performance and reduces energy over CPU, GPU, Cognitive SSD, and DeepStore platforms.

Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark1 for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.

The advancements in learning-based embedding models have propelled the embedding vector to become a vital data structure for various semantic search applications. The graph-based vector search method, which relies on the k-nearest neighbor (k-NN) graph, has gained immense popularity due to its superior performance in recall and throughput compared to other vector search methods. However, the construction of the k-NN graph is a time-consuming process, which not only affects the efficiency of identifying the optimal k-NN graph but also impacts data updates, especially for large-scale vector datasets. Despite the availability of various software-level optimization methods, such as iterative, refinement, and incremental methods, their peak performance is still limited to the underlying CPU platform. Furthermore, the existing GPU and specialized accelerator solutions only focus on partial k-NN graph construction methods and still encounter low efficiency when executing iterative methods due to irregular memory access, low data reuse, and redundant distance calculations.

Graph is widely utilized as a key data structure in many applications such as social network and recommendation systems. However, many real-world graphs are constructed with large-scale correlated data, which typically involves massive random memory accesses and distance calculation resulting in considerable processing time and energy consumption on CPUs and GPUs. In this work, we present GCiM, a specialized processing-in-memory architecture for efficient graph construction and updating. By directly deploying the computing units on the logic layer of the 3D stacked memory, GCiM benefits from memory-level parallelism and further improves the memory access efficiency with both optimized processing ordering and data layout. In addition, we notice that the computing engines for graph updating suffer from dramatic utilization imbalance and integrate a power gating module to cut down the power supply of the idle computing engines at runtime and further enhance the energy efficiency. According to our experiments, GCiM shows 634.64X and 56.27X speedup while consuming 1194.14X and 505.07X less energy compared to CPU and GPU respectively.

DNN-based video object detection (VOD) powers autonomous driving and video surveillance industries with rising importance and promising opportunities. However, adversarial patch attack yields huge concern in live vision tasks because of its practicality, feasibility, and powerful attack effectiveness. This work proposes Themis, a software/hardware system to defend against adversarial patches for real-time robust VOD. We observe that adversarial patches exhibit extremely localized superficial feature importance in a small region with nonrobust predictions, and thus propose the adversarial region detection algorithm for adversarial effect elimination. Themis also proposes a systematic design to efficiently support the algorithm by eliminating redundant computations and memory traffics. Experimental results show that the proposed methodology can effectively recover the system from the adversarial attack with negligible hardware overhead.

ReRAM-based Computing-in-Memory (CiM) architecture has been considered a promising solution to high-efficiency neural network accelerator, by conducting in-situ matrix multiplications and eliminating the movement of neural parameters from off-chip memory to computing units. However, we observed specific features of Graph Convolutional Network (GCN) tasks pose design challenges to implement a high-efficiency ReRAM GCN accelerator. The ultra-large input feature data in some GCN tasks incur massive data movements, the extremely sparse adjacency matrix and input feature data involve the valid computation, and the super-large adjacency matrix that exceeds available ReRAM capacity causes frequent expensive write operations. To address the above challenges, we propose TARe, a Task-Adaptive CiM architecture, which consists of a hybrid in-situ computing mode to support the input feature in crossbar computing, a compact mapping scheme for efficient sparse matrix computing, and a write-free mapping to eliminate write activities in the computations with the super-large adjacency matrix. Additionally, TARe is facilitated with a task adaptive selection algorithm to generate optimized design schemes for graph neural network tasks that have various operand sizes and data sparsity. We evaluate TARe on 11 diverse graph neural network tasks and compare it with different design counterparts, and the results show that achieves 168.06× speedup and 10.95× energy consumption reduction on average over the baseline in common graph convolutional network workloads.

Efficient deep learning models, especially optimized for edge devices, benefit from low inference latency to efficient energy consumption. Two classical techniques for efficient model inference are lightweight neural architecture search (NAS), which automatically designs compact network models, and quantization, which reduces the bit-precision of neural network models. As a consequence, joint design for both neural architecture and quantization precision settings is becoming increasingly popular. There are three main aspects that affect the performance of the joint optimization between neural architecture and quantization: quantization precision selection (QPS), quantization aware training (QAT), and neural architecture searching (NAS). However, existing works focus on at most twofold of these aspects, and result in secondary performance. To this end, we proposed a novel automatic optimization framework, DAQUDAQU is an ancient liquor fermentation process., that allows jointly searching for Pareto-optimal neural architecture and quantization precision combination among more than 1047 quantized subnet models. To overcome the instability of the conventional automatic optimization framework, DAQU incorporates a warm-up strategy to reduce the accuracy gap among different neural architectures, and a precision-transfer training approach to maintain flexibility among different quantization precision settings. Our experiments show that the quantized lightweight neural networks generated by DAQU consistently outperform state-of-the-art NAS and quantization joint optimization methods.

Process discovery aims to extract descriptive process models from event logs. To date, various process discovery algorithms have been proposed for different application settings. However, most of them meet challenges in handling event logs produced from hierarchical multi-instance business processes, in which multiple sub-process instances are invoked by the execution of a parent process. To address the problem, a novel approach is presented to support the discovery of hierarchical multi-instance process models. Specifically, taking event logs with multi-instance information as input, the detailed implementation of our method can be generally divided into four steps: nesting relation detection, hierarchical event log construction, sub-process case identification, and hierarchical multi-instance model discovery. We have implemented our approach properly as plugins in the openly accessible ProM toolkit, and compared its performance against the state-of-the-art process discovery approaches over six publicly available event logs. Based on the experimental result, it is demonstrated that the proposed approach can effectively discover hierarchical multi-instance process models with better quality.

The amalgamation of artificial intelligence with Internet of Things (AIoT) devices have seen a rapid surge in growth, largely due to the effective implementation of deep neural network (DNN) models across various domains. However, the deployment of DNNs on such devices comes with its own set of challenges, primarily related to computational capacity, storage, and energy efficiency. This survey offers an exhaustive review of techniques designed to accelerate DNN inference on AIoT devices, addressing these challenges head-on. We delve into critical model compression techniques designed to adapt to the limitations of devices and hardware optimization strategies that aim to boost efficiency. Furthermore, we examine parallelization methods that leverage parallel computing for swift inference, as well as novel optimization strategies that fine-tune the execution process. This survey also casts a future-forward glance at emerging trends, including advancements in mobile hardware, the co-design of software and hardware, privacy and security considerations, and DNN inference on AIoT devices with constrained resources. All in all, this survey aspires to serve as a holistic guide to advancements in the acceleration of DNN inference on AIoT devices, aiming to provide sustainable computing for upcoming IoT applications driven by artificial intelligence.

With some specific characteristics such as elastics and scalability, cloud computing has become the most promising technology for online business nowadays. However, how to efficiently perform real-time job scheduling in cloud still poses significant challenges. The reason is that those jobs are highly dynamic and complex, and it is always hard to allocate them to computing resources in an optimal way, such as to meet the requirements from both service providers and users. In recent years, various works demonstrate that deep reinforcement learning (DRL) can handle real-time cloud jobs well in scheduling. However, to our knowledge, none of them has ever considered extra optimization opportunities for the allocated jobs in their scheduling frameworks. Given this fact, in this work, we introduce a novel DRL-based preemptive method for further improve the performance of the current studies. Specifically, we try to improve the training of scheduling policy with effective job preemptive mechanisms, and on that basis to optimize job execution cost while meeting users' expected response time. We introduce the detailed design of our method, and our evaluations demonstrate that our approach can achieve better performance than other scheduling algorithms under different real-time workloads, including the DRL approach.

SIMD extensions are widely adopted in multi-core processors to exploit data-level parallelism. However, when co-running workloads on different cores, compute-intensive workloads cannot take advantage of the underutilized SIMD lanes allocated to memoryintensive workloads, reducing the overall performance. This paper proposes Occamy, a SIMD co-processor that can be shared by multiple CPU cores, so that their co-running workloads can spatially share its SIMD lanes. The key idea is to enable elastic spatial sharing by dynamically partitioning all the SIMD lanes across different workloads based on their phase behaviors, so that each workload may execute in variable-length SIMD mode. We also introduce an Occamy compiler to support such variable-length vectorization by analyzing such phase behaviors and generating the vectorized code that works with varying vector lengths. We demonstrate that Occamy can improve SIMD utilization, and consequently, performance over three representative SIMD architectures, with negligible chip area cost.

Deep neural networks have shown outstanding performance on complex tasks. Recently, various researches have been developed to pursue fast and energy-efficient deep learning accelerators. However, devices may suffer from hard defects and hardware variability during its lifetime, which poses severe challenges to deep learning accelerators. To protect edge deep learning accelerators from fault-induced failures, we leverage the adversarial deep learning technique to tailor a lightweight online fault detection method for neural network accelerator chips. The proposed Adversarial Testing scheme (AT) is a function-level testing method outcompeting conventional test in several ways: negligible run-time overhead, super sensitivity to subtle hardware variations, which reduces chip over-kills and also the unnecessary diagnosis operations. The evaluation results show that AT can accurately detect fault occurrence and ensure the normal use of deep learning accelerator during its lifetime.

Solving systems of linear equations is a fundamental problem in scientific computing, which has been extensively researched for decades. One of the most well-known solvers is Algebraic Multigrid (AMG), which is widely used in high performance computing due to its good scalability. But currently accelerating AMG relies on the traditional von Neumann architecture of storage and computation separation, which leads to a large data transmission overhead. In this work, we propose a ReRAM-based processing-in-memory (PIM) architecture named AmgR, which overcomes the limitations of the traditional von Neumann architecture for AMG acceleration.However, accelerating AMG on ReRAM is non-trivial, because (1) AMG has many computing kernels of various types; (2) there are irregular operations that cannot be directly performed using matrix-vector multiplication suitable for ReRAM, i.e., aggregation operation; (3) ReRAM has poor write endurance, and a lot of data during AMG acceleration needs to be rewritten into ReRAM, resulting in high write cost. To address these issues, firstly, we propose a flexible architecture, which can realize each kernel of AMG and is reused by many kernels to improve resource utilization. Secondly, we propose a dedicated unit to realize the aggregation operation. Finally, we present a new mapping strategy to greatly reduce the number of data handling and writes. The experimental results show that the performance of AmgR is improved by an average of one and two orders of magnitude compared to HYPRE on the CPU and AmgX on the GPU, respectively, while the energy consumption is reduced by an average of two and three orders of magnitude.

Application-specific instruction-set processors (ASIP) has been widely used to speedup specific applications based on general-purpose processor (CPU) ISA-extension and scalar/vector units customization. However, as deep neural processing unit (NPU) becomes a dominant IP in nowadays system-on-chip (SoC) designs, the rich computational and memory resource of the NPUs integrated into advanced CPUs should also be utilized to achieve an even better application performance boost than vector/scalar compute-unit customization only. In this paper, we propose APPEND, a novel framework that tries to enrich the ASIP design methodology by taking the co-designing of both NPU and RISC-V CPU into consideration. To fully utilize and customize the resources of CPU and NPU, APPEND automatically (1) identifies the NPU-compatible kernels from the target application and partition the applications in between the NPU and RISC-V CPU core, (2) based on the application performance specification, applies the necessary hardware parameterization and customization based on the RISC-V CPU and NPU templates, and also (3) generates the extended NPU instructions to accelerate the critical and compatible kernels of the target application.

In this work, we propose Layer-Puzzle, a multi-task allocation and scheduling framework for multi-core NPUs. Based on the proposed latency-prediction model and dynamic parallelization scheme, Layer-Puzzle can generate near-optimal results for each layer under given hardware resources and traffic congestion levels. As an online scheduler, Layer-Puzzle performs a QoS-aware and dynamic scheduling method that picks the superior version from the previously compiled results and co-runs the selected tasks to improve system performance. Our experiments on MLPerf show that Layer-Puzzle can achieve up to 1.61X, 1.53X, and 1.95X improvement in ANTT, STP, and PE utilization, respectively.

This work proposes a ReRAM-based Computing-in-Memory (CIM) architecture for Neural Architecture Search (NAS) acceleration, ENASA, so that the compute-intensive NAS technology can be applied to various edge devices to customize the most suitable individual solution for their cases. In the popular one-shot NAS process, the system must repetitively evaluate the sampled sub-network within a large-scale supernet before converging to the best sub-network architecture. Thereby, how to map these iterative network inference tasks onto the CIM arrays makes a big difference in system performance. To realize efficient in-memory supernet sampling and evaluation, we design a novel mapping method that tactically executes a group of sub-nets in the CIM arrays, not only to boost the sub-net concurrency but also to eliminate the repetitive operations shared by these subnets. Meanwhile, to further enhance the subnet-level operation concurrency and sharing in the CIM arrays, we also tailor a novel CIM-friendly one-shot NAS algorithm that purposely samples those operation-sharing subnets in each iteration while still maintaining the convergence performance of NAS. According to the experimental results, our CIM NAS accelerator achieves an improvement of 196.6× and 1200× in performance speedup and energy saving respectively compared to the CPU+GPU baseline.

The attention mechanism is becoming an integral part of modern neural networks, bringing breakthroughs to Natural Language Processing (NLP) applications and even Computer Vision (CV) applications. Unfortunately, the superiority of attention mechanism comes from its ability to model relations between any two positions in long sequence, which incurs high inference overhead. For state-of-the-art AI workloads such as Bert or GPT-2, attention mechanism is reported to account up to 50% of the inference overhead. Previous works seek to alleviate this performance bottleneck by removing useless relations for each position and accelerate position-specific operations. However their attempts require selecting from a sequence of relations once for each position, which is essentially frequent on-the-fly pruning and breaks the inherent parallelism in attention mechanism. In this paper, we propose CTA, an algorithm-architecture co-designed solution that can substantially reduce theoretic complexity of attention mechanism, enabling significant speedup and energy saving. Inspired by the fact that the feature sequence encoded by attention mechanism contain a large number of semantic feature repetition, we propose a novel approximation scheme that can efficiently remove that repetition, only calculating attention among necessary features thus reducing computation complexity quadratically. To utilize this algorithmic bonus and empower high performance attention mechanism inference, we devise specialized architecture to efficiently support the proposed approximation scheme. Extensive experiments show that, on average, CTA achieves 27.7× speedup, 634.0× energy savings with no accuracy loss, and 44.2× speedup, 950.0× energy savings with around 1% accuracy loss over Nvidia V100-SXM2 GPU. Also, CTA achieves 22.8× speedup, 479.6× energy savings over ELSA accelerator+GPU system.

Compared to conventional neural networks, training a supernet for Neural Architecture Search (NAS) is very time consuming. Although current works have demonstrated that parallel computing can significantly speed up the training process, almost all of their parallelism still follow the conventional data- and model-based paradigms, which actually face performance issues in both computation and inter-node communication of the supernet training. To further improve the performance of current methods, we discover the unique path-parallelism that exists in supernet training, and proposed a novel training approach designed specifically for supernet. In detail, we focus on analyzing path correlations between subnets in a supernet and exploiting effective path-merging methods to reduce redundant computations and communications raised by concurrent subnets. Moreover, we also try to combine the proposed path parallelism with traditional intra-subnet parallelism to perform multi-level parallelization to further optimize the parallel performance. We present the detailed design and implementation of our method, and our experimental results show that our proposed approach can achieve up to 3.2x end-to-end speedup over conventional parallel training solutions, and 1.46x–5.78x speedup compared to the state-of-art supernet training frameworks.

Mixed-precision neural networks (MPNNs) that enable the use of just enough data width for a deep learning task promise significant advantages of both inference accuracy and computing overhead. FPGAs with fine-grained reconfiguration capability can adapt the processing with distinct data width and models, and hence, can theoretically unleash the potential of MPNNs. Nevertheless, commodity DPUs on FPGAs mostly emphasize generality and have limited support for MPNNs especially the ones with lower data width. In addition, primitive DSPs in FPGAs usually have much larger data width than that is required by MPNNs and haven't been sufficiently co-explored with MPNNs yet. To this end, we propose an open source MPNN accelerator design framework specifically tailored for FPGAs. In this framework, we have a systematic DSP-packing algorithm to pack multiple lower data width MACs in a single primitive DSP and enable efficient implementation of MPNNs. Meanwhile, we take DSP packing efficiency into consideration with MPNN quantization within a unified neural network architecture search (NAS) framework such that it can be aware of the DSP overhead during quantization and optimize the MPNN performance and accuracy concurrently. Finally, we have the optimized MPNN fine-tuned to a fully pipelined neural network accelerator template based on HLS and make best use of available resources for higher performance. Our experiments reveal the resulting accelerators produced by the proposed framework can achieve overwhelming advantages in terms of performance, resource utilization, and inference accuracy for MPNNs when compared with both handcrafted counterparts and prior hardware-aware neural network accelerators on FPGAs.

Quantum circuit simulation (QCS) is essential in the noisy intermediate scale quantum (NISQ) era when real quantum computers are scarce. However, fully tracking the states of a quantum system in QCS is highly challenging due to the exponential memory growth that significantly limits the computational reach of classical systems for QCS. Though it is straightforward to leverage secondary storage to extend the scale of QCS, excessive data movement between memory and storage dominates the simulation time, making this solution unrealistic. To tackle this challenge, we identify an intrinsic property of QCS and implement an open-source framework to effectively reduce data movement by >116x. We evaluate the framework on various benchmarks and demonstrate 4x memory reduction with only <20% overhead. On a memory constrained system, we show that it extends the scale of QCS to 32 qubits (64 GB memory requirement) while existing simulators are bounded to 28 qubits (4 GB memory requirement). Our implementation can be accessed via https://github.com/Zhaoyilunnn/qdao.

Graph convolutional neural network (GCN) extends deep learning to process graph data and demonstrates superior performance. However, due to the irregularity, graphs show inconsistent patterns across different regions, which leads to distinctions in data reusability and edge processing activity, and consequently poses impacts on hardware efficiency and resource utility. Prior accelerators seldom explore the distinct patterns across graph regions and adopt a fixed strategy for the whole graph without consideration for region-specific characteristics. In this paper, we identify the inconsistent patterns of graphs and characterize the distinctions between graph regions. Then, we propose an adaptive dataflow to adapt the region-specific patterns. Third, we implement PANG, a pattern-aware accelerator that can dynamically adjust the dataflow to exploit the reusability and alleviate the frequent destination switching. Evaluated on real-world datasets, PANG achieves significant performance improvement.

It has recently become trendy for neural network accelerators to scale from single-core to chiplet-based multichip architecture, as the growth of neural network depth and complexity are calling for the promotion of computation and memory capabilities. However, the unintended extensive inter-chip communication of chiplet-based accelerator may bottleneck the parallelism of deep learning inference, which is undesirable for many real-time applications and energy-efficient devices. Although it is imperative for novel schemes to be devised to alleviate this problem, related works are scarce. In this work, we present CampQ, a fine-grained communication-aware mixed-precision quantization method to accelerate inference parallelization by reducing the major inter-chiplet communication overhead. By leveraging the AutoML technique, CampQ is capable of determining different bit-width to activation groups according to thier transmission distances in on-package network. The experimental results show 1.4×-2.6× performance benefits and 29%-60% energy reduction over the 16-bit models for various neural networks and parallelism approaches.

Recently, video processing tasks, such as video enhancement and analysis, have received increasing attention from both academics and industries. However, the current video processing procedure on edge decouples the decoding phase and the subsequent video processing tasks, missing the opportunity to accelerate the procedure by orchestrating video decoding and enhancement stages. Thus, we propose an intelligent video processing workflow and architecture(IVP) for cloud-edge video streaming. For edge devices that receive compressed videos, IVP can perform direct DNN-based video enhancement, e.g., super-resolution and frame-interpolation. By leveraging the metadata motion vectors and residuals extracted from the encoded video, our architecture will significantly eliminate unnecessary frame pixels being processed by the DNNs and improve execution efficiency. The proposed IVP and workflow are proved to reduce up to 90% of the processing latency while producing accurate and high-quality videos. Furthermore, we observe a significant portion of similar optical flow in time domain of continuous videos, which can be used to reduce the computation overhead. Thus, to utilize such temporal similarity of optical flow, the proposed IVP is upgraded to be capable of reusing previous computation results, which further improves energy efficiency of the whole system.

The emerging Resistive RAM (ReRAM) technology significantly boosts the performance and the energy efficiency of the deep learning accelerators (DLAs) via the Computing-in-Memory (CiM) architecture. However, ReRAM-based DLA also suffers a high occurrence rate of memory faults. How to detect and protect against the faults in ReRAM devices poses great challenges to ReRAM-based DLA design. In this work, we propose RRAMedy, an in-situ fault detection and network remedy framework for ReRAM-based DLAs. With the proposed Adversarial Example Testing, which is a lifetime on-device and on-line fault detection technique, it achieves high detection coverage of both hard faults and soft faults at a low run-time cost. In addition, it employs an edge-cloud collaborative model retraining method to tolerate the detected faults by leveraging the inherent fault-adaptive capability of DNNs. Meanwhile, to enable in-situ model remedy when the cloud assistance is absent due to security or overhead issues, we propose to accelerate the fault-masking retraining process on edge devices with parallelized Knowledge Transfer. Our experimental results show that the proposed fault detection technique achieves high fault detection accuracy and delivers real-time testing performance. Meanwhile, the proposed retraining approach greatly alleviates the accuracy degradation problem and achieves excellent performance speedups over the baselines.

Soft errors in large VLSI circuits have a significant impact on computing- and memory-intensive neural network (NN) processing. Understanding the influence of soft errors on NNs is critical to protect against soft errors for reliable NN processing. Prior work mainly relies on fault simulation to analyze the influence of soft errors on NN processing. They are accurate but usually specific to limited configurations of errors and NN models due to the prohibitively slow simulation speed especially for large NN models and datasets. With the observation that the influence of soft errors propagates across a large number of neurons and accumulates as well, we propose to characterize the soft error-induced data disturbance on each neuron with a normal distribution model using the central limit theorem and develop a series of statistical models to analyze the behavior of NN models under soft errors in general. The statistical models reveal not only the correlation between soft errors and the accuracy of NN models but also how NN parameters, such as quantization and architecture affect the reliability of NNs. The proposed models are compared with fault simulations and verified comprehensively. In addition, we observe that the statistical models that characterize the soft error influence can also be utilized to predict fault simulation results in many cases and we explore the use of the proposed statistical models to accelerate fault simulations of NNs. Our experiments show that the proposed accelerated fault simulation provides almost two orders of magnitude speedup with negligible loss of simulation accuracy compared to the baseline fault simulations.

The RRAM-based neuromorphic computing system (NCS) has amassed explosive interests for its superior data processing capability and energy efficiency than traditional architectures, and thus being widely used in many data-centric applications. The reliability and security issues of the NCS, therefore, become an essential problem. In this article, we systematically investigated the adversarial threats to the RRAM-based NCS and observed that the RRAM hardware feature can be leveraged to strengthen the attack effect, which has not been granted sufficient attention by previous algorithmic attack methods. Thus, we proposed two types of hardware-aware attack methods with respect to different attack scenarios and objectives. The first is an adversarial attack, VADER, which perturbs the input samples to mislead the prediction of neural networks. The second is fault injection attack, EFI, which perturbs the network parameter space such that a specified sample will be classified to a target label, while maintaining the prediction accuracy on other samples. Both attack methods leverage the RRAM properties to improve the performance compared with the conventional attack methods. Experimental results show that our hardware-aware attack methods can achieve nearly 100% attack success rate with extremely low operational cost, while maintaining the attack stealthiness.

Network sparsity or pruning is a pivotal technology for edge intelligence. Resistive random access memory (RRAM)-based accelerators, featuring dense storage and processing in memory capability, have demonstrated the superior computing performance and energy efficiency over the traditional CMOS-based accelerators for neural network applications. Unfortunately, RRAM-based accelerators suffer the performance or energy degradation when deploying pruned models, impairing their competition in the edge intelligence scenarios. We observed the essential reason is the pruning technology and the mapping strategy in prior RRAM-based accelerator and are optimized individually. As a result, the random zeros in the pruned deep neural network are irregularly distributed in the crossbars, rendering the degradation of computation parallelism of the crossbar without crossbar demand reduction. In this work, we propose a coordinated model pruning and mapping framework to jointly optimize of model accuracy and efficiency of RRAM-based accelerators. As for the mapping, we first decouple weight matrices in the bit-wise way and map the bit matrices to different crossbars, where the signed weights are represented with the two’s complement so as that save half desired crossbars. As for the pruning, we prune weight bits at the crossbar granularity so that free the crossbars holding the pruned bits. Furthermore, we employ an reinforcement learning (RL) approach to automatically select the optimal crossbar-aware bit-pruning strategy for any given neural network without laborious human efforts. We conducted the experiments on a set of representative neural networks and compared our framework with the state-of-the-art (SOTA) bit-sparsity works. The results show that automatic structured bit-pruning saves up to 89.64% energy reduction and 84.12% area overhead compared to existing PRIME-like architecture. Besides, our framework outperforms the SOTA bit-sparsity design by$1.5\times $in terms of the energy reduction on the RRAM-based accelerator.

Bit-serial architectures (BSAs) are becoming increasingly popular in low-power neural network processor (NNP) designs for edge scenarios. However, the performance and energy efficiency of state-of-the-art BSA NNPs heavily depends on both the proportion and distribution of ineffectual weight bits in neural networks (NNs). To boost the performance of typical BSA accelerators, we present Bit-Pruner, a software approach to learn BSA-favored NNs without resorting to hardware modifications. Bit-Pruner not only progressively prunes but also restructures the nonzero bits in weights so that the number of nonzero bits in the model can be reduced and the corresponding computing can be load-balanced to suit the target BSA accelerators. On top of Bit-Pruner, we further propose a Pareto frontier optimization algorithm to adjust the bit-pruning rate across network layers and fulfill diverse NN processing requirements in terms of performance and accuracy for various edge scenarios. However, an aggressive Bit-Pruner can lead to nontrivial accuracy loss, especially for lightweight NNs and complex tasks. To this end, the alternating direction method of multipliers (ADMMs) is adapted to the retraining phase in Bit-Pruner to smooth the abrupt disturbance due to bit-pruning and enhance the resulting model accuracy. According to the experiments, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation and achieves an optimized tradeoff between NN accuracy and energy efficiency even under very-aggressive performance constraints. When pruned models are deployed onto typical BSA accelerators, the average performance is$2.1\times $and$1.6\times $higher than the baseline networks without pruning and those with classical weight pruning, respectively.

The reduction of neural parameters and operations for the applications on embedded and IoT platforms in current deep neural network (DNN) architectures has received increasing attention. Relatively, the intermediate feature maps of such lightweight neural networks begin to grow and usually outsize the on-chip memory as the new bottleneck, which introduces considerable power-consuming off-chip memory accesses. To reduce the feature-induced memory accesses, operator fusion has been proposed to parallelize the execution of multiple convolutional layers and shown significant reduction of off-chip memory accesses. However, how to fuse the neural operators is still a challenging issue that heavily depends on both the neural network (NN) topology and the specific DNN accelerator configuration. In this work, we observed prior operator fusion approaches fail to guarantee memory-level optimality as they search in the constrained operator fusion design space. Considering the complexity of the NN topologies and the constrained resources of the DNN accelerators, we develop a novel operator fusion framework, Optimus. Optimus includes an accurate memory cost model dedicated to the scheduler to evaluate the potential operator-fusion schemes and a directed acyclic graph-based operator fusion algorithm for both off-line and on-line workload deployment scenarios, which altogether generates high-efficiency operator-fusion solutions for arbitrary network models running on DNN accelerators. The experimental results show that Optimus reduces 17–75% off-chip memory accesses and obtains 1.86×–3.66× energy efficiency on state-of-the-art DNN workloads when compared to the baselines and brings significant power-efficiency boost to the DNN accelerators of different architectures and dataflows.

The efficiency of deep neural network (DNN) solutions on real hardware devices are mainly decided by the DNN architecture and the compiler-level scheduling strategy on the hardware. When we try to fully exploit the underlying hardware and obtain the optimal tradeoff between DNN accuracy and runtime performance, we discovered that the two optimization goals of DNN architecture and scheduling policy are intimately related to each other. However, current hardware-aware Neural Architecture Search (NAS) methods primarily focus on the DNN architecture search process, ignoring the effects of various compiler-level scheduling strategies (e.g., graph-level optimization, loop transformations, parallelization, etc.) on network candidates being evaluated in the search process. As a result, they may overlook the true-optimal DNN implementations on hardware, which can only be discovered by trying-out different combinations of scheduling strategies and DNN architectures. This work proposes a NAS framework (CHaNAS) that searches for not only the network architecture but also the dedicated compiler-level scheduling policy, as the optimal co-design solution on the target hardware. We propose to use a block-based pre-scheduling methodology to reduce the co-design search space and enable the automatic generation of the optimal co-design, including the network architecture and the tensor programs that practice the scheduling policy. Further, we introduce a new search objective function based on the generalization gap to prevent the selection of architectures that are prone to overfitting. We evaluate CHaNAS on Imagenet on different hardware back-ends against the state-of-the-art hardware-aware search method based on the MobileNet-v3 search space. Experimental results show that the co-design solutions obtained by ChaNAS show up to 1.6×, 1.9×, and 1.7×, 24 performance boost on NVIDIA P100 GPU, Intel Xeon 8163 CPU, and Samsung Note 10 Mobile, respectively, over the baselines of the same-level accuracy.

How to efficiently process concurrent data tasks such as online analytical queries in datacenter environments is still a big challenge for current computing techniques. One of the fundamental reasons is that their task execution normally involves large numbers of distributed data operators, which are always expensive in terms of communication time. To improve the general performance, various advanced approaches on the execution optimization of data operators have been proposed in the past years. However, most of them focus on application-level optimization, such as using data locality scheduling to reduce network traffic. Moreover, few of them has considered the optimization opportunities for concurrent execution of multiple data operators. In this paper, we propose a novel coflow-based scheduling system called CoFlop, which aims to improve network communication time for multiple distributed operators at a query level, and on that basis to lay a solid foundation for the development of a network-aware query execution system in datacenter networks. We introduce the detailed system design of CoFlop and conduct a simulation-based evaluation with large concurrent distributed join operations. Compared to existing methods, the experimental results show that CoFlop can perform better in the presence of different large workloads.

Deformable convolution networks (DCNs) proposed to address image recognition with geometric or photometric variations typically involve deformable convolution that convolves on arbitrary locations of input features. The locations change with different inputs and induce considerable dynamic and irregular memory accesses that cannot be handled by classic neural network accelerators (NNAs). Moreover, bilinear interpolation (BLI) operation, which is required to obtain deformed features in DCNs, also cannot be deployed on existing NNAs directly. Although a general purposed processor (GPP) seated along with classic NNAs can process the deformable convolution, the processing on GPP can be extremely slow due to the limited parallel computing capability and massive additional data movement. To address the problem, we develop a DCN accelerator on existing NNAs to support both the standard convolution and deformable convolution. Specifically, for the dynamic and irregular accesses in DCNs, we have both the input and output features divided into tiles and build a tile dependency table (TDT) to track the irregular tile dependency at runtime. With the TDT, we further develop an on-chip tile scheduler to handle the dynamic and irregular accesses efficiently. In addition, we propose a novel mapping strategy to enable parallel BLI processing on NNAs and apply layer fusion techniques for more energy-efficient DCN processing. According to our experiments, the proposed accelerator achieves orders of magnitude higher performance and energy efficiency compared to the typical computing architectures including ARM, ARM+TPU, and GPU with 6.6% chip area penalty to a classic NNA.

Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution (WG-Conv) in improving neural network (NN) fault tolerance. Based on the observation, we evaluate WG-Conv fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of WG-Conv for cost-effective NN protection against soft errors. Specifically, we mainly investigate how WG-Conv can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), fault-aware retraining, and constrained activation functions. According to our experiments, WG-Conv can reduce the fault-tolerant design overhead by 55.77% on average without any accuracy loss compared to standard convolution (ST-Conv), and further reduce the computing overhead by 17.24% when the inherent fault tolerance of WG-Conv is considered. When it is applied on fault-tolerant NNs enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in the presence of various faults.

Vision transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs’ works mainly optimize performance and accuracy, but ViTs’ reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix–matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault-tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs’ accuracy significantly with minor computing overhead in the presence of various soft errors.

ReRAM-based accelerators have shown great potential in neural network acceleration via in-memory analog computing. However, high-precision analog-to-digital converters (ADCs), which are required by the ReRAM crossbars to achieve high-accuracy network model inference, play an essential role in the energy-efficiency of the accelerators. Based on the discovery that the ADC precision requirements of crossbars are different, we propose the model-aware crossbarwise ADC precision assignment and the accompanied information-lossless low-bit ADCs to reduce energy overhead without sacrificing model accuracy. In experiments, the proposed information-lossless ReRAM accelerator, InfoX, only consumes 8.97% ADC energy of the SOTA baseline with no accuracy degradation at all.

Real-time ego-motion tracking and 3D structural estimation are the fundamental tasks for the ubiquitous cyper-physical systems, and they can be conducted via the state-of-the-art Edge-Based Visual Odometry (EBVO) algorithm. However, the intrinsic data-intensive process of EBVO emplaces a memory-wall hurdle in practical deployment on conventional von-Neumann-style computing systems. In this work, we attempt to leverage SRAM based processing-in-memory (PIM) technique to alleviate such memory-wall bottleneck, so as to optimize the EBVO systematically from the perspectives of the algorithm layer and physical layer. In the algorithm layer, we first investigate the data reuse patterns of the essential computing kernels required for the feature detection and pose estimation steps in EBVO, and propose PIM friendly data layout and computing scheme for each kernel accordingly. We distill the basic logical and arithmetical operations required in the algorithm layer, and in the physical layer, we propose a novel bit-parallel and reconfigurable SRAM-PIM architecture to realize the operations with high computing precision and throughput. Our experimental result shows that the proposed multi-layer optimization allows for high tracking accuracy of EBVO, and it can improve 11x processing speed and reduce 20x energy consumption compared to the CPU implementation.

Graph-based vector search that finds best matches to user queries based on their semantic similarities using a graph data structure, becomes instrumental in data science and AI application. However, deploying graph-based vector search in production systems requires high accuracy and cost-efficiency with low latency and memory footprint, which existing work fails to offer. We present VStore, a graph-based vector search solution that collaboratively optimizes accuracy, latency, memory, and data movement on large-scale vector data based on in-storage computing. The evaluation shows that VStore exhibits significant search efficiency improvement and energy reduction while attaining accuracy over CPU, GPU, and ZipNN platforms.

Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation. Other than the computing efficiency, we observe its great potential in improving NN fault tolerance and evaluate its fault tolerance comprehensively for the first time. Then, we explore the use of fault tolerance of winograd convolution for either fault-tolerant or energy-efficient NN processing. According to our experiments, winograd convolution can be utilized to reduce fault-tolerant design overhead by 27.49% or energy consumption by 7.19% without any accuracy loss compared to that without being aware of the fault tolerance.

Network-on-Chips (NoCs) have been viewed as a promising alternative to traditional on-chip communication architecture for the increasing number of IPs in modern chips. To support the vast design space exploration of application-specific NoC characteristics with arbitrary topologies, in this paper, we propose a fast estimation framework to predict power, performance, and area (PPA) of NoCs based on graph neural networks (GNNs). We present a general way of modeling the application and the NoC with user-defined parameters as an attributed graph, which can be learned by the GNN model. Experimental results show that on the unseen realistic applications, the proposed method achieves the accuracy of 97.36% on power estimation, 97.83% on area estimation, and improves the accuracy of the network-level and system-level performance predictor over the topology-constrained baseline method by 6.52% and 4.73% respectively.

Hardware accelerators based on systolic arrays have become the dominant method for efficient processing of deep neural networks (DNNs). Although such designs provide significant performance improvement compared to its contemporary CPUs or GPUs, their power efficiency and area efficiency are greatly limited by the large computing array and on-chip memory. In this work, we demonstrate that we can further improve the efficiency of systolic accelerators using emerging carbon nanotube field-effect transistors (CNFETs) by stacking the computing logic and on-chip memory on multiple layers and utilizing monolithic 3D (M3D) vias for low-latency communication. We comprehensively explore the design space and present MOCCA, the first process variation tolerable CNFET-based systolic DNN accelerator. We validate MOCCA against previous 2D accelerators on state-of-the-arts DNN models. On average, MOCCA achieves the same throughput with 6.12× and 2.12× improvement respectively on performance and power efficiency in a 2× reduced chip footprint.

2.5D chiplet technology is gaining popularity for the efficiency of integrating multiple heterogeneous dies or chiplets on interposers, and it is also considered an ideal option for agile silicon system design by mitigating the huge design, verification, and manufacturing overhead of monolithic SoCs. Although it significantly reduces development costs by chiplet reuse, the design and fabrication of interposers also introduce additional high non-recurring engineering (NRE) costs and development cycles which might be prohibitive for application-specific designs having low volume.

Memory sub-system including the data memory and instruction memory in a typical CGRA often takes up considerable chip area and even dominates the performance of CGRAs. Observing the common phenomenon that instruction memory in CGRAs is highly under-utilized while the data memory is over-committed or vice versa, we propose a CGRA with unified memory architecture, UM-CGRA, to enable flexible on-chip memory sharing between data and instructions. Furthermore, PEs are also augmented to share data between neighbors working in parallel. Also, an on-chip memory sharing-aware mapping algorithm is developed to unleash the potential of the proposed architecture. Our experimental results show that UM-CGRA achieves 77% performance improvement on average over the baseline CGRA given the same amount of total on-chip memory. When setting the same performance goal, UM-CGRA achieves 10.7% chip area-saving and 28.6% energy efficiency improvement on average.

Although systolic accelerators have become the dominant method for executing Deep Neural Networks (DNNs), their performance efficiency (quantified as Energy-Delay Product or EDP) is limited by the capabilities of silicon Field-Effect Transistors (FETs). FETs constructed from Carbon Nanotubes (CNTs) have demonstrated > 10 × EDP benefits, however, the processing variations inherent in carbon nanotube FETs (CNFETs) fabrication compromise the EDP benefits, resulting > 40% performance degradation. In this work, we study the impact of CNT process variations and present Canopy, a process variation aware systolic DNN accelerator by leveraging the spatial correlation in CNT variations. Canopy co-optimizes the architecture and dataflow to allow computing engines in a systolic array run at their best performance with non-uniform latency, minimizing the performance degradation incurred by CNT variations. Furthermore, we devise Canopy with dynamic reconfigurability such that the microarchitectural capability and its associated flexibility achieves an extra degree of adaptability with regard to the DNN topology and processing hyper-parameters (e.g., batch size). Experimental results show that Canopy improves the performance by 5.85 × (4.66 ×) and reduces the energy by 34% (90%) when inferencing a single (a batch of) input compared to the baseline design under an iso-area comparison across seven DNN workloads.

The growing complexity and diversity of deep neural network (DNN) applications have inspired intensive research on specialized DNN accelerators and also the design automation frameworks. Previous specialized NN acceleratos roughly fall into two categories of implementation, either the no-pipelined architecture that relies on a generic processing unit (PU) to sequentially execute the DNN layers in a layer-wise way, or the fully-pipelined architecture that dedicates interconnected customized PUs to the corresponding DNN layers in the model. Thus, such designs often suffer from either the resource under-utilization issue faced by no-pipelined accelerators or the resource scalability problem brought by the over-deep pipeline designs. In this work, we propose a novel class of design solution for DNN acceleration, segment-grained pipeline architecture (SPA). In the SPA accelerator, the targeted workload of DNN models will be divided into many segments and each segment will be sequentially executed on the shared interconnected PUs in a pipeline manner, so that they will benefit from both the efficiency of pipelined execution and also the flexibility of sharing PUs across different model layers. Particularly, we found that the efficiency of the implemented SPA accelerator significantly depends on the segmentation strategies of the models and the hardware resources assignment policy for PUs. Therefore, we introduce an automated design framework, AutoSeg, that includes a parameterized SPA accelerator template and a co-design engine that will generate the efficient model segmentation solution and hardware pipeline design parameters for the acceleration workload. Experimental results show that the SPA solutions generated by the AutoSeg framework achieve $1.2\times to 6.3\times$ speedup when compared to ASIC-based general DNN processors, and the FPGA designs implemented by AutoSeg also achieve as high as $3.4\times$ DSP efficiency and $3.6\times$ throughput improvement.

In DNN processors, main memory consumes much more energy than arithmetic operations. Therefore, many memory-oriented network scheduling (MONS) techniques are introduced to exploit on-chip data reuse opportunities and reduce accesses to memory. However, to derive the theoretical lower bound of memory overhead for DNNs is still a significant challenge, which also sheds light on how to reach memory-level optimality by means of network scheduling. Prior work on MONS mainly focused on disparate optimization techniques or missed some of the data reusing opportunities in diverse network models, thus their results are likely to deviate from the true memory-optimality that can be achieved in processors. This paper introduces Olympus, which comprehensively considers the entire memory-level DNN scheduling space, formally analyzes the true memory-optimality and also how to reach the memory-optimal schedules for an arbitrary DNN running on a DNN processor. The key idea behind Olympus is to derive a true memory lower-bound regarding both the intra-layer and inter-layer reuse opportunities, which has not been simultaneously explored by prior works. Evaluation on SOTA DNN processors of different architectures shows that Olympus can guarantee the minimum off-chip memory access, and it reduces 12.3-85.6% DRAM access and saves 7.4-70.3% energy on the latest network models.

Real-time inference of deep learning models on embedded and energy-efficient devices becomes increasingly desirable with the rapid growth of artificial intelligence on edge. Specifically, to achieve superb energy-efficiency and scalability, efficient parallelization of single-pass deep neural network (DNN) inference on chip multiprocessor (CMP) architectures is urgently required by many time-sensitive applications. However, as the number of processing cores scales up and the performance of cores has grown much fast, the on-chip inter-core data movement is prone to be a performance bottleneck for computation. To remedy this problem and further improve the performance of network inference, in this work, we introduce a communication-aware DNN parallelization technique called CAP, by exploiting the elasticity and noise-tolerance of deep learning algorithms on CMP. Moreover, in the hope that the conducted studies can provide new design values for real-time neural network inference on embedded chips, we also have evaluated the proposed approach on both multi-core Neural Network Accelerators (NNA) chips and general-purpose chip-multiprocessors. Our experimental results show that the proposed CAP can achieve 1.12×-1.65× system speedups and 1.14×-2.70× energy efficiency for different neural networks while maintaining the inference accuracy, compared to baseline approaches.

Dynamic resources management in reconfigurable processors often manifests as a hard online decision-making task, which should yield premier solutions that must meet Quality-of-Service (QoS) requirements while maximizing the system’s efficiency. Most prior works rely on a hard-to-train predictor to model the complicated relationships between processor configurations and performance. To decide the proper resource allocation, the predictor needs to tentatively evaluate a group of possible configurations, and then decide the best configuration for the workload. This tedious process has an expensive runtime overhead for resource configuration in processors. Besides, prior works focus on improving the prediction accuracy, however, higher performance prediction cannot guarantee a good system outcome. Inspired by recent advances in adversarial learning, we present a generative adversarial network (GAN)-based framework, Amphis, which can directly generate the on-demand processor configuration for any scheduled-in application. By evaluating Amphis on a reconfigurable processor with 18 different workloads, our results demonstrate that the GAN-based method provides tremendous overhead reduction (up to 90%) compared to the SOTA prediction-based method WNNM while providing higher resource utilization.

In-memory computing (IMC) is recognized as one of the most promising architecture solution to realize energy-efficient neural network inference. Amongst many memory technology, resistive RAM (RRAM) is a very attractive device to implement the IMC-based neural network accelerator architecture, which is particularly suitable for power-constrained IoT systems. Due to the nature of low leakage and in-situ computing, the dynamic power consumption of dot-production operations in RRAM crossbars dominates the chip power, especially when applied to low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a state-aware RRAM accelerator (SARA) architecture for energy-efficient low-precision neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the RRAM accelerator can perform in a lower power state. Furthermore, we also leverage the proposed RRAM accelerator architecture to reduce the power consumption of high-precision network inference with both single-level or multilevel RRAM. The evaluation results show that for binary neural networks, our design saves 40.53% RRAM computing energy on average over the baseline. For high precision neural networks, the proposed method reduces 11.67% computing energy on average without any accuracy loss.

With the advancement of deep neural networks (DNNs), the applications driven by DNNs have been spread from the cloud to the edge. However, the intensive computations and data movements in CNNs impede the adoption of DNNs in resource-constraint edge devices.Quantization, a common model compression method, has attracted a lot of attention as it enables efficient inference by lowering the data bit-width of CNN parameters. Due to the features of massive storage and computing-in-memory array, resistive memory (RRAM) has established the energy efficiency and small area processing-in-memory (PIM) for the acceleration of DNNs at the edge end. However, when deploying the network onto resistive-memory-based PIM (RRAM-based PIM), there will be tremendous unused cells due to the mismatch between the structure of the neural network layer and memory array, resulting in the resource under-utilization and low computation efficiency. In this work, we observed prior quantization approaches fail to improve hardware resource utilization as they ignored the hardware structure information in RRAM. Thus, combining the information of the neural network model and hardware information is essential for a high-utilization RRAM-based PIM design. Considering the vast model parameters and heterogeneous RRAM crossbar structure, we develop a novel quantization framework by leveraging the AutoML technique, i.e., RaQu, which automatically generates a fine-grained quantization strategy for any model that fully utilizes the resource of RRAM-based PIM. The experimental results show that RaQu achieves at most 29.2%–37.4% and 1.8%–3.3% improvement in resource utilization and model accuracy, respectively, compared to prior coarse-grained quantization methods.

Hardware faults on the regular 2-D computing array of a typical deep learning accelerator (DLA) can lead to dramatic prediction accuracy loss. Prior redundancy design approaches typically have each homogeneous redundant processing element (PE) to mitigate faulty PEs for a limited region of the 2-D computing array rather than the entire computing array to avoid the excessive hardware overhead. However, they fail to recover the computing array when the number of faulty PEs in any region exceeds the number of redundant PEs in the same region. The mismatch problem deteriorates when the fault injection rate rises and the faults are unevenly distributed. To address the problem, we propose a hybrid computing architecture (HyCA) for fault-tolerant DLAs. It has a set of dot-production processing units (DPPUs) to recompute all the operations that are mapped to the faulty PEs despite the faulty PE locations. According to our experiments, HyCA shows significantly higher reliability, scalability, and performance with less chip area penalty when compared to the conventional redundancy approaches. Moreover, by taking advantage of the flexible recomputing, HyCA can also be utilized to scan the entire 2-D computing array and detect the faulty PEs effectively at runtime.

Due to the nonvolatility nature of resistive RAM (ReRAM), dynamic operations in the arrays contribute to a much larger portion of power in ReRAM-based neural networks than static power. To reduce the dynamic power ofin-situoperations with neural parameters, precision-tuning is considered a viable approach of approximate computing to tradeoff excessive computation exactness for power and efficiency gains. However, the switching overhead of precision tuning in hardware severely impacts its effectiveness when the systems need to quickly react to the change of environment, user constraint or input quality. This work for the first time investigates the feasibility of agile precision tuning for neural network accelerators to benefit from approximate computing. The proposed computing in memory (CiM) CNN accelerators fully utilize the normally off characteristics of memristor crossbars to achieve instant network precision tuning without worrying about the model reloading penalty. The ReRAM-based accelerator, with the proposed neural parameter mapping policy and the novel mixed-model training method, induces negligible precision-switching latency and power consumption when compared with traditional variable precision accelerators. In evaluation with state-of-the-art workloads, the proposed ReRAM deep learning and neural network architecture saves 58.3%–62.47% area overhead over the baseline design. We also leverage the proposed ReRAM accelerator architecture to build a novel always-on key-word spotting (KWS) system. The KWS design can switch between different precision modes to capture the relevant sound with high accuracy. The experimental results show the precision-adjustable KWS architecture saves considerable operating energy when fed with realistic test-sets of audio data.

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can alleviate the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which is able to predict results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstreams, Alchemist contributes to a remarkable performance speedup of 5x with negligible accuracy loss. Nevertheless, we still find that the original compressed video coded by standard algorithms such as H.264 is not suitable to be directly manipulated, due to diverse compressed structures. Although obviating the requirement to recover all RGB frames, the accelerator must parse the entire compressed video bitstream to locate reference frames and extract useful metadata. If we combine the video codec with the proposed compressed video analysis, additional optimizations can be obtained. Therefore, to cope with the mismatch between current video coding algorithms, such as H.264 and neural network-based video analysis, we propose a specialized coding strategy to generate compressed video bitstreams more suitable for transmission and analysis, which further simplifies the decoding stage of video analysis and is capable of achieving significant storage reduction.

As a part of Advanced Driver Assistance Systems (ADASs), Consensus-based Speed Advisory Systems (CSAS) have been proposed to recommend a common speed to a group of vehicles for specific application purposes, such as emission control and energy management. With Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I) technologies and advanced control theories in place, state-of-the-art CSAS can be designed to get an optimal speed in a privacy-preserving and decentralized manner. However, the current method only works for specific cost functions of vehicles, and its execution usually involves many algorithm iterations leading long convergence time. Therefore, the state-of-the-art design method is not applicable to a CSAS design which requires real-time decision making. In this article, we address the problem by introducing MPC-CSAS, a Multi-Party Computation (MPC) based design approach for privacy-preserving CSAS. Our proposed method is simple to implement and applicable to all types of cost functions of vehicles. Moreover, our simulation results show that the proposed MPC-CSAS can achieve very promising system performance in just one algorithm iteration without using extra infrastructure for a typical CSAS.

Carbon nanotube field-effect transistors (CNFETs) emerge as a promising alternative to CMOS transistors for the much higher speed and energy efficiency, which makes the technology particularly suitable for building the energy-hungry last-level cache (LLC). However, the process variations (PVs) in CNFET caused by the imperfect fabrication lead to large timing variations, and the worst case timing dramatically limits the LLC operation speed. Particularly, we observe that the CNFET-based cache latency distribution is closely related to the LLC layouts. For the two typical LLC layouts that have the CNT growth direction aligned to the cache way direction and cache set direction, respectively, we proposed variation-aware set aligned (VASA) cache and variation-aware way aligned (VAWA) cache in combination with corresponding cache optimizations, such as data shuffling and page mapping to enable low-latency cache for frequently used data. According to our experiments, the optimized LLC reduces the average access latency by 32% and 45% compared to the baseline designs on the two different CNFET layouts, respectively, while it improves the overall performance by 6% and 9%, and reduces the energy consumption by 4% and 8%, respectively. In addition, with both the architecture-induced latency variation and PV-incurred latency variation considered in a unified model, we extended the VAWA and VASA cache designs for the CNFET-based NUCA, and the proposed NUCA achieves both significant performance improvement and energy saving compared to the straightforward variation-aware NUCA.

Adversarial attacks have been viewed as the primary threat to the security of neural networks. Hence, extensive adversarial defense techniques have been proposed to protect the neural networks from adversarial attacks, allowing for the application of neural networks to the security-sensitive tasks. Recently, the emerging devices, e.g., Resistive RAM (RRAM), attracted extensive attention for establishing the hardware platform for neural networks to tackle the inadequate computing capability of the traditional computing platform. Though the emerging devices exhibit the instinct instability issues due to the advanced manufacture technology, including hardware variations and defects, the error-resilience capability of neural networks enables the wide deployment of neural networks on the emerging devices. In this work, we find that the natural instability in emerging devices impairs the security of neural networks. Specifically, we design an enhanced adversarial attack, Variation-oriented ADvERsarial (VADER) attack which leverages the inherent hardware variations in RRAM chips to penetrate the protection of adversarial defenses and mislead the prediction of neural networks. We evaluated the effectiveness of VADER across various protected neural network models and the result shows that VADER achieves higher success attack rate over other adversarial attacks.

This work proposes an intelligent video processing architecture for bandwidth-efficient edge-cloud video streaming. On receiving the bandwidth-saving low-quality video streaming in compressed format, the proposed architecture can perform direct DNN-based video enhancement, e.g., super-resolution and motion-compensated frame interpolation (MCFI), on streams. By utilizing the metadata motion vectors and residuals extracted from the encoded video, our workflow will significantly eliminate the unnecessary pixels being processed by the video-enhancing DNNs, and greatly promote the execution efficiency. The evaluation results on popular datasets show that our architecture can reduce the edge-side processing latency of video-enhancing DNNs by 90% compared to the traditional flow while producing accurate and high-quality videos on edge.

Graph is widely utilized as a key data structure in many applications like social network and recommendation systems. However, real-world graph construction typically involves massive random memory accesses and distance calculation, resulting in considerable processing time and energy consumptions on CPUs and GPUs. In this work, we present GCiM, a specialized processing-in-memory architecture for efficient graph construction and update. By directly deploying the computing units on the logic layer of the 3D stacked memory, GCiM benefits from memory-level parallelism and further improves the memory access efficiency with both optimized processing ordering and data layout. According to our experiments, GCiM shows 634.64X and 53.29X speedup while consuming 1470.7X and 442.56X less energy compared to CPU and GPU respectively.

ReRAM-based Computing-in-Memory (CiM) architecture has been considered an ideal solution to neural networks, by conducting in-situ matrix multiplications without moving the neural parameters from memory cells. However, we found that keeping the parameters static in ReRAM cells, i.e. weight-static processing, is not the sole choice to implement emerging graph neural networks (GNNs) that operate on the input of ultra large graphs. Therefore, we propose TARe, a Task-Adaptive CiM architecture that supports multiple different in-situ computing modes for Graph Learning. With the proposed novel hybrid in-situ computing architecture, TARe achieves 451.98× speedup on average over the baseline in SOTA GNN workloads.

Network sparsity or pruning is an extensively studied method to optimize the computation efficiency of deep neural networks (DNNs) for CMOS-based accelerators, such as FPGAs and GPUs. Though the RRAM-based accelerator has demonstrated superior performance and energy efficiency for DNN tasks, deploying the sparse neural networks desires dedicated consideration to save resource consumption without introducing the expensive index overhead and sophisticated control. To exploit the potential of sparse neural network design on the RRAM-based accelerator, we propose an automatic structured bit-pruning design, ASBP, to harmonize the optimization objective of DNN sparsity with efficient RRAM deployment. Specifically, ASBP prunes the bits of weight which are split into different crossbars and thus, free the zero-value crossbar when mapping the neural network into RRAM-based accelerators without extra hardware modification. Meanwhile, ASBP employs the reinforcement learning (RL) approach to automatically select the best crossbar-aware bit-sparsity strategy for any given neural network without laborious human efforts. According to our experiments on a set of representative neural networks, ASBP saves up to 79.01% energy consumption and 54.79% area overhead compared to the baseline that deploys the original DNN on the RRAM-based accelerator. Besides, ASBP outperforms the state-of-the-art bit-sparsity design by 1.4x in terms of the energy reduction on the RRAM-based accelerator.

Pixel-level data redundancy in video induces additional memory and computing overhead when neural networks are employed to mine spatiotemporal patterns, e.g. activity and event labels from video streams. This work proposes PixelSieve, to enable highly efficient CNN-based activity analysis directly from video data in compressed formats. Instead of recovering original RGB frames from compressed video, PixelSieve utilizes the built-in metadata in compressed video streams to distill only the critical pixels that render relevant spatiotemporal features, and then conducts efficient CNN inference with the condensed inputs. PixelSieve removes the overhead of video decoding and significantly improves the performance of CNN-based video analysis by 4.5x on average.

Chiplet based multi-die integration has been thought as a key enabler of the agile chip development flow. For 2.5D based multi-die system, Network on Interposer plays an essential role in the performance and the development cost of the chips. This work proposed a reusable NoI design for agile AI chip customization. The proposed NoI design can self-adapt to the inter-die communication patterns of various neural network applications, so the produced interposers can be reused across different AI chip specifications. Experimental results show the proposed NoI design brings 42.7%$\sim$79.5% of total data communication latency reduction in different scenarios, and it also decreased the area overhead by 26.4%.

The recent proposed Deformable Convolutional Networks (DCNs)greatly enhance the performance of conventional Convolutional Neural Networks (CNNs) on vision recognition tasks by allowing flexible input sampling during inference runtime. DCNs introduce an additional convolutional layer for adaptive sampling offset generation, followed by a bilinear interpolation (BLI) algorithm to integerize the generated non-integer offset values. Finally, a regular convolution is performed on the loaded input pixels. Compared with conventional CNNs, DCN demonstrated significantly increased computational complexity and irregular input-dependentmemory access patterns, making it a great challenge for deploying DCNs onto edge devices for real-time computer vision tasks. In this work, we propose RECOIN, a processing-in-memory (PIM) architecture, which supports DCN inference on resistive memory (ReRAM)crossbars, thus making the first DCN inference accelerator possible. We present a novel BLI processing engine that leverage both row-and column-oriented computation for in-situ BLI calculation. Amapping scheme and an address converter are particular designed to accommodate the intensive computation and irregular data access. We implement the DCN inference in a 4-stage pipeline and evaluate the effectiveness of RECOIN on six DCN models. Experimental results show RECOIN achieves respectively 225×and 17.4×improvement in energy efficiency compared to general-purpose CPU and GPU. Compared to two state-of-the-art ASIC accelerators, RECOIN achieve 26.8× and 20.4× speedup respectively.

As neural networks (NNs) are being widely deployed in many cloud-oriented systems for safety-critical tasks, the privacy and security of NNs become significant concerns to users in the cloud platform that shares the computation infrastructure such as memory resource. In this work, we observed that the memory timing channel in the shared memory of cloud multi-core architecture poses the risk of network model information leakage. Based on the observation, we propose a learning-based method to steal the model architecture of the NNs by exploiting the memory timing channel without any high-level privilege or physical access. We first trained an end-to-end measurement network offline to learn the relation between memory timing information and NNs model architecture. Then, we performed an online attack and reconstructed the target model using the prediction from the measurement network. We evaluated the proposed attack method on a multi-core architecture simulator. The experimental results show that our learning-based attack method can reconstruct the target model with high accuracy and improve the adversarial attack success rate by 42.4%.

Multi-task learning systems are commonly adopted in many real-world AI applications such as intelligent robots and self-driving vehicles. Instead of improving single-network performance, this work proposes a specialized Multi-Task Deep Learning Accelerator architecture, MT-DLA, to improve the performance of concurrent networks by exploiting the shared feature and parameters across these models. It is shown in our evaluation with realistic multi-task workloads, MT-DLA dramatically eliminates the memory and computation overhead caused by the shared parameters, activations and computation result. In the experiments with real-world multi-task learning workloads, MT-DLA brings about 1.4x-7.0x energy efficiency boost when compared to the baseline neural network accelerator without multi-task support.

Ego-motion estimation with 3D perception using visual odometry (VO) is known to be robust and economical among the existing odometry techniques. However, existing VO solutions are typically both computation intensive and memory intensive, which dramatically inhibits their deployment in IoT platforms such as robotic vehicles and handheld devices mostly equipped with resource-constrained MCU-level processors. To enable real-time and high-quality VO on these scenarios with thrifty resource budgets, we investigate state-of-the-art edge-based VO (EBVO) and propose an optimization framework called PicoVO that can greatly reduce the amount of computation as well as the memory footprint from the perspectives of both algorithm and implementation. First of all, we revisit the key processing stages of EBVO and propose an EBVO-oriented lightweight edge detector in the pre-processing stage, a sparse-to-dense processing scheme in the tracking stage, and a lightweight key-frame management in the post-processing stage. In addition to the algorithmic optimization, we further develop a dedicated quantization scheme particularly for the 3D feature calculation and Levenberg-Marquardt (LM) solver that are critical to the memory footprint and computation requirements of PicoVO. Evaluation on realistic RGB-D benchmark datasets is conducted on NUCLEO-F767ZI equipped with a 216MHz Cortex-M7 MCU and 512KB RAM. It reveals that PicoVO achieves 33fps@320x240 with high tracking precision comparable to state-of-the-art VOs on PC.

Neural network search (NAS) projects a promising direction to automate the design process of efficient and powerful neural network architectures. Nevertheless, the NAS techniques have to dynamically generate a large number of candidate neural networks, and iteratively train and evaluate these on-line generated network architectures, thus they are extremely time-consuming even when deployed on large GPU clusters, which dramatically hinders the adoption of NAS. Though recently there are many specialized architectures proposed to accelerate the training or inference of neural networks, we observe that existing neural network accelerators are typically targeted at static neural network architectures, and they are not suitable to accelerate the evaluation of the dynamical neural network candidates evolving during the NAS process, which cannot be deployed onto current accelerators via the off-line compilation.To enable rapid and energy-efficient NAS in compact single-chip solutions, we propose NASA, a specialized architecture for one-shot based NAS acceleration. It is able to generate, schedule, and evaluate the candidate neural network architectures for the target machine learning workload with high speed, significantly alleviating the processing bottleneck of one-shot NAS. Motivated by the observation that there are considerable computation sharing opportunities among the different neural network candidates generated in one-shot NAS, NASA is equipped with an on-chip network fusion unit to remove the redundant computation during the network mapping stage. In addition, the NASA accelerator can partition and re-schedule the candidate neural network architectures at fine-granularity to maximize the chance of data reuse and improve the utilization of the accelerator arrays integrated to accelerate network evaluation. According to our experiments on multiple one-shot NAS tasks, NASA achieves 33.52× performance speedup and 214.33× energy consumption reduction on average when compared to aCPU-GPU system.

Neural network layer fusion has been proposed to parallelize the inference of neural layers and thus significantly reduces the feature-induced memory accesses. However, how to fuse the neural layers is still a challenging issue that heavily depends on both the network architecture and the specific DNN processor configuration. This work formalizes the layer fusion problem for DNN processors, proves that prior fusion solutions cannot guarantee memory-level optimality, and presents a novel neural network fusion framework, Optimus. Optimus includes an accurate memory cost model to evaluate fusion schemes, and a Computing-Graph (CG) based layer fusion algorithm, which generates high-efficiency layer-fusion schemes for arbitrary network architectures on DNN processors. The proposed off-line and on-line graph-based fusion algorithms can reduce 10.1% - 72.2% off-chip memory traffic and obtain 1.71x - 3.94x energy efficiency over SOTA baselines on DNN workloads, and they bring significant power-efficiency boost to the DNN processors of different architectures and dataflows.

Automatically design an efficient DNN solution for a given deep learning task on the target hardware mainly decided by the neural network architecture and the schedule mapping strategy, where the two goals are closely coupled with each other to fully exploit the advantages of the underlying hardware. Prior hardware-aware Neural Architecture Search (NAS) methods mostly ignore the impacts of different scheduling policies (e.g., graph-level optimization, loop transformations, parallelization, etc.) on network candidates being evaluated in the search process. Thus, they may miss the true-optimal architecture that can only be discovered by trying-out different scheduling policies. This work proposes a NAS framework (CHaNAS) that searches for not only the network architecture but also the dedicated scheduling policy, as the optimal co-design solution on target hardware that fully exploits the advantages of the underlying hardware. We propose to use a block-based pre-scheduling methodology to reduce the co-design search space, and enable the automatic generation of the optimal co-design, including the network architecture and the tensor programs that practice the scheduling policy. We evaluate CHaNAS on Imagenet on different hardware back-ends against the state-of-the-art hardware-aware search method MobileNet-v3. Experimental results show that the co-design solutions obtained by ChaNAS show up to 1.6x, 1.9x, and 1.7x performance boost on NVIDIA P100 GPU, Intel Xeon 8163 CPU, and Samsung Note 10 Mobile, respectively, over the baselines of the same-level accuracy.

This special session focuses on test for artificial intelligence (AI) chips, with important issues from design for test (DFT) to on-line testing. The first talk discusses different DFT implementations and their tradeoffs as well as test access and configuration infrastructure for AI chips with many cores. The second talk discusses low-cost on-line fault detection and hardware salvaging techniques for neural network processors. The last talk gives case studies for testing industrial AI SOC chips, with an emphasis on the automatic test pattern generation (ATPG) methodology.

Graph neural networks (GNNs) emerge as a powerful approach to process non-euclidean data structures and have been proved powerful in various application domains such as social networks and e-commerce. While such graph data maintained in real-world systems can be extremely large and sparse, thus employing GNNs to deal with them requires substantial computational and memory overhead, which induces considerable energy and resource cost on CPUs and GPUs. In this article, we present a specialized accelerator architecture, EnGN, to enable high-throughput and energy-efficient processing of large-scale GNNs. The proposed EnGN is designed to accelerate the three key stages of GNN propagation, which is abstracted as common computing patterns shared by typical GNNs. To support the key stages simultaneously, we propose the ring-edge-reduce(RER) dataflow that tames the poor locality of sparsely-and-randomly connected vertices, and the RER PE-array to practice RER dataflow. In addition, we utilize a graph tiling strategy to fit large graphs into EnGN and make good use of the hierarchical on-chip buffers through adaptive computation reordering and tile scheduling. Overall, EnGN achieves performance speedup by 1802.9X, 19.75X, and 2.97X and energy efficiency by 1326.35X, 304.43X, and 6.2X on average compared to CPU, GPU, and a state-of-the-art GCN accelerator HyGCN, respectively.

3D convolutional neural networks (CNNs) are gaining increasing popularity in the area of video-based action/activity analysis. Compared to 2D convolutions that share the filters in a 2D spatial domain, 3D convolutions further reuse filters in the temporal dimension to capture temporal-domain features in the video. How to exploit the data locality in the temporal dimension directly impacts the energy efficiency of specialized architectures for 3D CNN inference. Prior works on specialized 3D-CNN accelerators employ additional on-chip memories and multicluster architecture to reuse data among the process element (PE) arrays, which is very expensive for low-power chip implementation. Instead of harvesting in-memory data locality, we propose the architecture of systolic cube to exploit the spatial and temporal localities in 3D CNNs, which moves the reusable data in-between PEs connected via a 3D-cube network-on-chip. Furthermore, due to the existence of visual feature reappearance in the temporal domain, there exists a considerable portion of repetitive pixels and activations among the feature maps captured at adjacent time slots. To eliminate such temporal redundancy in 3D CNNs, the proposed accelerator architecture is equipped with a redundancy detection and elimination mechanism, capable of skipping the computations with the same activations and parameters when reusing the convolutional filters along the temporal dimension. In our evaluation, the experimental results show that the systolic-cube architecture contributes to a considerable energy-efficiency boost for state-of-the-art activity-recognition benchmarks and datasets.

Stereo vision is widely deployed on robots and drones to enable depth estimation at a low cost. The combination of lightweight deep neural network (DNN) and cost volumes algorithm is proved to possess the advantages of both high depth estimation accuracy and speed. However, currently there is no accelerator architecture compatible with both efficient DNN inference and cost generation algorithms such as stereo matching. This work proposes a stereo vision accelerator called Dadu-eye, dedicated to real-time processing of high-resolution image streams. The proposed architecture adopts a pipelined hardware design with the techniques of operation approximation and scheduling-level optimization. First, a cost estimation block is designed to generate cost volumes from both luminance and color information. Second, a super pipelined multiplication and accumulation array with a row scan-based fused-layer convolution scheduling is proposed to perform the encoding and decoding neural network efficiently. Finally, an optical flow block is designed and cooperates with the array to approximately predict half of the frames’ depth to achieve real-time (30fps) processing on 1080p view. Based on the SMIC 40 nm CMOS process, this stereo vision accelerator achieves 5.3 TOPS/W power efficiency and significantly reduces 81% off-chip memory access.

Large data centers are currently the mainstream infrastructures for big data processing. As one of the most fundamental tasks in these environments, the efficient execution of distributed data operators (e.g., join and aggregation) are still challenging current data systems, and one of the key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer data locality optimization to reduce network traffic or on network-layer data flow optimization to increase bandwidth utilization. However, the techniques in the two layers are totally independent from each other, and performance gains from a joint optimization perspective have not yet been explored. In this article, we propose a novel approach called NEAL (NEtwork-Aware Locality scheduling) to bridge this gap, and consequently to further reduce communication time for distributed big data operators. We present the detailed design and implementation of NEAL, and our experimental results demonstrate that NEAL always performs better than current approaches for different workloads and network bandwidth configurations.

Artificial Intelligence of Things (AIoT) processors fabricated with newer technology nodes suffer rising soft errors due to the shrinking transistor sizes and lower power supply. Soft errors on the AIoT processors particularly the deep learning accelerators (DLAs) with massive computing may cause substantial computing errors. These computing errors are difficult to be captured by the conventional training on general-purposed processors such as CPUs and GPUs in a server. Applying the offline trained neural network models to the edge accelerators with errors directly may lead to considerable prediction accuracy loss. To address the problem, we propose a remote retraining framework (R2F) for remote AIoT processors with computing errors. It takes the remote AIoT processor with soft errors in the training loop such that the on-site computing errors can be learned with the application data on the server and the retrained models can be resilient to the soft errors. Meanwhile, we propose an optimized partial triple modular redundancy (TMR) strategy to enhance the retraining. According to our experiments, R2F enables elastic design tradeoffs between the model accuracy and the performance penalty. The top-5 model accuracy can be improved by 1.93%–13.73% with 0%–200% performance penalty at high fault error rate. In addition, we notice that the retraining requires massive data transmission and even dominates the training time and propose a sparse increment compression approach for the data transmission optimization, which reduces the retraining time by 38%–88% on average with negligible accuracy loss over straightforward remote retraining.

Prior works typically conducted the fault analysis of neural network accelerator computing arrays with simulation and focused on the prediction accuracy loss of the neural network models. There is still a lack of systematic fault analysis of the neural network acceleration system that considers both the accuracy degradation and system exceptions, such as system stall and running overtime. To that end, we implemented a representative neural network accelerator and corresponding fault injection modules on a Xilinx ARM-FPGA platform and evaluated the reliability of the system under different fault injection rates when a series of typical neural network models are deployed on the neural network acceleration system. The entire fault injection and reliability evaluation system is open-sourced on GitHub. With comprehensive experiments on the system, we identify the system exceptions based on the various abnormal behaviors of the FPGA-based neural network acceleration system and analyze the underlying reasons. Particularly, we find that the probability of the system exceptions dominates the reliability of the system. The faults also incur accuracy degradation of the neural network models, but the influence depends on the applications of the models and can vary greatly. In addition, we also evaluated the use of conventional triple modular redundancy (TMR) and demonstrated the challenge of TMR with both experiments and analytical models, which may shed light on the reliability design of the FPGA-based neural network acceleration system.

The increasing hardware failures caused by the shrinking semiconductor technologies pose substantial influence on the neural accelerators and improving the resilience of the neural network execution becomes a great design challenge especially to mission-critical applications such as self-driving and medical diagnose. The reliability analysis of the neural network execution is a key step to understand the influence of the hardware failures, and thus is highly demanded. Prior works typically conducted the fault analysis of neural network accelerators with simulation and concentrated on the prediction accuracy loss of the models. There is still a lack of systematic fault analysis of the neural network acceleration system that considers both the accuracy degradation and system exceptions such as system stall and early termination.In this work, we implemented a representative neural network accelerator and fault injection modules on a Xilinx ARM-FPGA platform and conducted fault analysis of the system using four typical neural network models. We had the system open-sourced on github. With comprehensive experiments, we identify the system exceptions based on the various abnormal behaviours of the FPGA-based neural network acceleration system and analyze the underlying reasons. Particularly, we find that the probability of the system exceptions dominates the reliability of the system and they are mainly caused by faults in the DMA, control unit and instruction memory of the accelerators. In addition, faults in these components also incur moderate accuracy degradation of the neural network models other than the system exceptions. Thus, these components are the most fragile part of the accelerators and need to be hardened for reliable neural network execution.

Sparsification is an efficient solution to reduce the demand of on-chip memory space for deep convolutional neural networks (CNNs). Most of state-of-the-art CNN accelerators can deliver high throughput for sparse CNNs by searching pairs of nonzero weights and activations, and then sending them to processing elements (PEs) for multiplication-accumulation (MAC) operations. However, their PE scales are difficult to be increased for superior and efficient computing because of the significant internal interconnect and memory bandwidth consumption. To deal with this dilemma, we propose a sparsity-aware architecture, called Swan, which frees the search process for sparse CNNs under limited interconnect and bandwidth resources. The architecture comprises two parts: a MAC unit that can free the search operation for the sparsity-aware MAC calculation, and a systolic compressive dataflow that well suits the MAC architecture and greatly reuses inputs for interconnect and bandwidth saving. With the proposed architecture, only one column of the PEs needs to load/store data while all PEs can operate in full scale. Evaluation results based on a place-and-route process show that the proposed design, in a compact factor of 4096 PEs, 4.9TOP/s peak performance, and 2.97W power running at 600MHz, achieves 1.5-2.1× speedup and 6.0-9.1× higher energy efficiency than state-of-the-art CNN accelerators with the same PE scale.

Resistive RAM (ReRAM) is a promising device to realize the Computing in Memory (CiM) architecture, suitable for power-constrained IoT systems. Because of low leakage, the dot-production operations in ReRAM crossbars dominate the chip power, especially when implementing low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a State-Aware ReRAM Accelerator (SARA) architecture for energy-efficient neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the ReRAM accelerator can perform in a lower-power state. The evaluation shows that our design reduces 47% energy over the baseline without compromising the network accuracy.

Convolutional neural networks (CNNs) have become the state-of-the-art technique in many classification tasks in IoT system. However, the low-power and area-constraint edge devices are unable to afford the expensive cost of CNNs. Resistive random access memory (RRAM) is attractive for establishing the CNN accelerator at the edge end due to the features of scalability, low-power and in-situ dot-product. However, mapping a random network architecture onto a general-purpose RRAM accelerator suffers a severe issue of resource underutilization. The neural network quantization offers an opportunity to rescue the degraded resource utilization. Selecting the bit-width for the vast parameters is impractically completed by human labor. This paper proposes an AutoML-based array-aware quantization and mapping framework that generates the fine-grained mixed-precision neural networks to optimize resource utilization in RRAM. In this framework, we design a two-stage learning and array-aware grouping strategy to quickly explore the huge searching space. The experimental results show that the proposed framework achieves 18.2%~36.1% improvement in resource utilization and 0.9%~3.3% increase in model accuracy over prior coarse-grained quantization methods.

Previous neural network accelerators tailored to video analysis only accept data of RGB/YUV domain, requiring decompressing the video that are often compressed before transmitted from the edge sensors. A compressed video processing accelerator can remove the decoding overhead, and gain performance speedup by operating on more compact input data. This work proposes a novel deep learning accelerator architecture, Alchemist, which predicts results directly from the compressed video bitstream instead of reconstructing the full RGB images. By utilizing the metadata of motion vector and critical blocks extracted from bitstream, Alchemist contributes to remarkable performance speedup of 5x with negligible accuracy loss.

Bit-serial architectures (BSAs) are becoming increasingly popular in low power neural network processor (NNP) design. However, the performance and efficiency of state-of-the-art BSA NNPs are heavily depending on the distribution of ineffectual weight-bits of the running neural network. To boost the efficiency of third-party BSA accelerators, this work presents Bit-Pruner, a software approach to learn BSA-favored neural networks without resorting to hardware modifications. The techniques proposed in this work not only progressively prune but also structure the non-zero bits in weights, so that the number of zero-bits in the model can be increased and also load-balanced to suit the architecture of the target BSA accelerators. According to our experiments on a set of representative neural networks, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation. When the bit-pruned models are deployed onto typical BSA accelerators, the average performance is 2.1X and 1.5X higher than the baselines running non-pruned and weight-pruned networks, respectively.

Carbon Nanotubu field-effect transistor (CNFET) that promises both higher clock speed and energy efficiency becomes an attractive alternative to the conventional power-hungry CMOS cache. We observe that the CNFET-based cache constructed with typical SRAM cells has distinct energy consumption when reading/writing 0 and 1 from/to it. For instance, the energy consumption of writing 1 to an SRAM cell is almost 10X higher than writing 0. With this observation, we propose an energy-efficient cache design called CNT-Cache to take advantage of this feature. It predicts the cache line access pattern based on the latest cache line access history. On top of the prediction, it decides the optimal cache line encoding to match the cache operation preferences at runtime. According to our experiments on a set of benchmark programs, the optimized CNFET-based D-Cache reduces the dynamic power consumption by 22.2% on average compared to the baseline CNFET cache.

DNN/Accelerator co-design has shown great potential in improving QoR and performance. Typical approaches separate the design flow into two-stage: (1) designing an application-specific DNN model with high accuracy; (2) building an accelerator considering the DNN specific characteristics. However, it may fails in promising the highest composite score which combines the goals of accuracy and other hardware-related constraints (e.g., latency, energy efficiency) when building a specific neural-network-based system. In this work, we present a single-stage automated framework, YOSO, aiming to generate the optimal solution of software-and-hardware that flexibly balances between the goal of accuracy, power, and QoS. Compared with the two-stage method on the baseline systolic array accelerator and Cifar10 dataset, we achieve 1.42x~2.29x energy or 1.79x~3.07x latency reduction at the same level of precision, for different user-specified energy and latency optimization constraints, respectively.

The design of neural network architecture for code approximation involves a large number of hyper-parameters to explore, it is a non-trivial task to find an neural-based approximate computing solution that meets the demand of application-specified accuracy and Quality of Service (QoS). Prior works do not address the problem of ‘optimal’ network architectures design in program approximation, which depends on the user-specified constraints, the complexity of dataset and the hardware configuration. In this paper, we apply Neural Architecture Search (NAS) for searching and selecting the neural approximate computing and provide an automatic framework that tries to generate the best-effort approximation result while satisfying the user-specified QoS/accuracy constraints. Compared with previous method, this work achieves more than 1.43x speedup and 1.74x energy reduction on average when applied to the AxBench benchmarks.

Processing-in-Memory (PIM) or Near-Data Processing has been recognized as the most potential solution to resolve the ever-aggravating memory wall especially as the thrive of memory-intensive scale-out workloads such as graph computing and data analytics. However, when the future computing system becomes more and more likely to adopt PIM architectures as a type of the storage and processing component, there is a lack of literature and research work on the general scheduling framework with the emerging heterogeneous system except for some ad-hoc task partitioning methods with specialized PIM designs. This work is the first to propose a formalized model to quantitatively describe the multi-task scheduling problem in PIM+CPU platform without loss of generality, and also an optimized task mapping-and-scheduling algorithm to boost the hardware utility for these novel heterogeneous systems. The proposed scheduling framework is fully aware of the data access bandwidth and processing capability distinction between the CPU and PIM devices, and also the implications of task mapping on the bandwidth contention, data communication intensity and hardware utility for the concurrent workloads. Experimental results show that, compared to the traditional scheduling algorithm for heterogeneous system, the proposed method is able to improve the system performance by over 10% and the energy efficiency by almost 10% for multi-core scale-out applications.

With the rapid progress of artificial intelligence (AI) algorithms, multi-modal deep neural networks (DNNs) have been applied to some challenging tasks, e.g., image and video description to process multi-modal information from vision and language. Resistive-memory-based processing-in-memory (ReRAM-based PIM) has been extensively studied to accelerate either convolutional neural network (CNN) or recurrent neural network (RNN). According to the requirements of their core layers, i.e. convolutional layers and linear layers, the existing ReRAM-based PIMs adopt different optimization schemes for them. Directly deploying multi-modal DNNs on the existing ReRAM-based PIMs, however, is inefficient because multi-modal DNNs have combined CNN and RNN where the primary layers differ depending on the specific tasks. Therefore, a high-efficiency ReRAM-based PIM design for multi-modal DNNs necessitates an adaptive optimization to the given network. In this work, we propose HitM, a high-throughput ReRAM-based PIM for multi-modal DNNs with a two-stage workflow, which consists of a static analysis and an adaptive optimization. The static analysis generates the layer-wise resource and computation information with the input multi-modal DNN description and the adaptive optimization produces a high-throughput ReRAM-based PIM design through the dynamic algorithm based on hardware resources and the information from the static analysis. We evaluated HitM using several popular multi-modal DNNs with different parameters and structures and compared it with a naïve ReRAM-based PIM design and an optimal-throughput ReRAM-based PIM design that assumes no hardware resource limitations. The experimental results show that HitM averagely achieves 78.01% of the optimal throughput while consumes 64.52% of the total hardware resources.

Building FPGA-based graph learning accelerators is very time-consuming due to the low-level RTL programming and the complicated design flow of FPGA development. It also requires the architecture and hardware expertise from the Graph Neural Network (GNN) application developers to tailor efficient accelerator designs on FPGAs. This work proposes an automation framework, DeepBurning-GL, which is compatible with state-of-the-art graph learning frameworks such as Deep Graph Library so that the developers can easily generate application-specific GNN accelerators from the software-described models. First, DeepBurning-GL employs a GNN performance analyzer to locate the performance bottleneck of specific GNN applications and decide the major design architectures and parameters that meet the user-specified constraints. Second, DeepBurning-GL provides a series of pre-built design templates such as computing templates and memory templates, which can be parameterized and fused to generate the final accelerator design. It also includes an optimizer that conducts automatic optimization by adjusting the accelerator architectural parameters. In evaluation, we use DeepBurning-GL to generate customized accelerators on three different FPGA platforms for various GNN models and workloads. The experimental results show that the generated accelerators achieve 179.4X and 40.1X energy-efficiency boost over the CPU and GPU solutions on average and deliver a 6.28X speedup and 6.73X energy-efficiency improvement on average compared to the latest GNN accelerator HyGCN on Alveo U50.

Deep Reinforcement Learning (DRL) is substantially resource-consuming, and it requires large-scale distributed computing-nodes to learn complicated tasks, like videogame and Go play. This work attempts to down-scale a distributed DRL system into a specialized many-core chip and achieve energy-efficient on-chip DRL. With the customized Network-on-Chip that handles the communication of on-chip data and control-signals, we proposed a Synchronous Asynchronous RL Architecture (SARLA) and the according many-core chip that completely avoids the unnecessary data duplication and synchronization activities in multi-node RL systems. In evaluation, the SARLA system achieves considerable energy-efficiency boost over the GPU-based implementations for typical DRL workloads built with OpenAI-gym.

Regular 2D computing array is widely utilized for the processing of the major neural network operations in many deep learning accelerators (DLAs). Hardware failures on the array can lead to considerable computing errors and prediction accuracy loss. Prior works proposed to add homogeneous redundant PEs to each row or column of the regular computing array to mitigate faulty PEs, but they may fail to recover the computing array from faults when the number of faulty PEs in a row or column exceeds the number of redundant PEs in the corresponding row or column. The problem gets worse when the faults are not evenly distributed across the computing array. To address the problem, we propose a hybrid computing architecture (HCA) for fault-tolerant DLAs. Instead of adding homogeneous redundant PEs to the regular computing array of DLAs, it has a dot-production processing unit (DPPU) to recompute the operations that are mapped to the faulty PEs concurrently without performance penalty under moderate fault injection. Even under high fault injection, HCA can be degraded smoothly and remains functional. In addition, DPPU exploits the parallelism within each operation and processes the network operations sequentially, so it can tolerate faulty PEs in arbitrary locations and ensures steady performance under distinct fault distributions. According to our experiments, HCA shows significantly higher reliability and performance under various fault injection with comparable chip area penalty compared to the conventional redundancy approaches.

Emerging memristor-based architectures are promising for data-intensive applications as these can enhance the computation efficiency, solve the data transfer bottleneck and at the same time deliver high energy efficiency using their normally-off/instant-on attributes. However, their storing devices are more susceptible to manufacturing defects compared to the traditional memory technologies because they are fabricated with new materials and require different manufacturing processes. Hence, in order to ensure correct functionalities for these technologies, it is necessary to have accurate fault modeling as well as proper test methodologies with high test coverage. In this paper, we propose technology specific cell-level defect modeling, accurate fault analysis and yield improvement solutions for memristor-based memory as well as Computation-In-Memory (CIM) architectures. Our overall contributions cover three abstraction levels, namely, device, architecture and system. First, we propose a device-aware test methodology in which we have introduced a key device-level characteristic to develop accurate defect model. Second, we demonstrate a yield analysis framework for memristor arrays considering reliability and permanent faults due to parametric variations and explore fault-tolerant solutions. Third, a lightweight on-line test and repair schemes is proposed for emerging CIM devices in machine learning applications.

Traditional memory technologies face severe challenges in meeting the ever-increasing power and memory bandwidth requirements for high-performance computing and big-data analyses. Several emerging memory technologies are promising as the replacements of SRAM or DRAM. Among them, STT-MRAM can be used to replace SRAM as the last-level cache (LLC). However, it suffers from high write energy and latency. In this article, we investigate data patterns written from SRAM-based upper-level cache to STT-MRAM-based LLC to explore the write energy reduction potential. Depending on the data layout within a cache line, redundant bits can be identified and eliminated from write back operations to save STT-MRAM write energy. We also propose a dynamic profiling method to accommodate different application characteristics. The extensive simulation results show that write energy can be saved by 37.05% ∼ 38.89% for static profiling and 19.76% ∼ 34.29% for dynamic profiling.

Generative neural network is a new category of neural networks and it has been widely utilized in many applications such as content generation, unsupervised learning, segmentation, and pose estimation. It typically involves massive computing-intensive deconvolution operations that cannot be fitted to conventional neural network processors directly. However, prior works mainly investigated specialized hardware architectures through intensive hardware modifications to the existing deep learning processors to accelerate deconvolution together with the convolution. In contrast, this article proposes a novel deconvolution implementation with a software approach and enables fast and efficient deconvolution execution on the existing deep learning processors. Our proposed method reorganizes the computation of deconvolution and allows the deep learning processors to treat it as the standard convolution by splitting the original deconvolution filters into multiple small filters. Compared to prior acceleration schemes, the implemented acceleration scheme achieves 2.4× -4.3× performance speedup and reduces the energy consumption by 27.7 -54.5 percent on a set of realistic benchmarks. In addition, we have also applied the deconvolution computing approach to the off-the-shelf commodity deep learning processors. The performance of deconvolution also exhibits significant performance speedup over prior deconvolution implementations.

As the speed gap of the modern processor and the off-chip main memory enlarges, on-chip cache capacity increases to sustain the performance scaling. As a result, the cache power occupies a large portion of the total power budget. Spin transfer torque magnetic memory (STT-MRAM) is proposed as a promising solution for the low power cache design due to its high integration density and ultralow leakage power. Nevertheless, the high write power and latency of STT-MRAM become new barriers for the commercialization of this emerging technology. In this paper, we investigate the thermal effect on the access performance of STT-MRAM, and observe that the temperature can affect the write delay and energy significantly. Then, we explore the nonuniform cache access (NUCA) design of the chip-multiprocessors with STT-MRAM-based last level cache (LLC). A thermal aware data migration policy, called “Thermosiphon,” which takes advantage of the thermal property of STT-MRAM, is proposed to reduce the LLC write energy. This policy splits the LLC into different regions dynamically based on the thermal distribution monitored by thermal sensors available on-chip, and adaptively migrates write intensive data among different thermal regions considering the thermal gradient. Compared to the conventional NUCA design, our proposed design can save 41.2% write energy at most and 13.01% on average with negligible hardware overhead.

With the advancements of neural networks, customized accelerators are increasingly adopted in massive AI applications. To gain higher energy efficiency or performance, many hardware design optimizations such as near-threshold logic or overclocking can be utilized. In these cases, computing errors may happen and the computing errors are difficult to be captured by conventional training on general purposed processors (GPPs). Applying the offline trained neural network models to the accelerators with errors directly may lead to considerable prediction accuracy loss. To address this problem, we explore the resilience of neural network models and relax the accelerator design constraints to enable aggressive design options. First of all, we propose to train the neural network models using the accelerators' forward computing results such that the models can learn both the data and the computing errors. In addition, we observe that some of the neural network layers are more sensitive to the computing errors. With this observation, we schedule the most sensitive layer to the attached GPP to reduce the negative influence of the computing errors. According to the experiments, the neural network models obtained from the proposed training outperform the original models significantly when the CNN accelerators are affected by computing errors.

This work investigates how to effectively train binarized neural networks (BNNs) for the specialized low-precision neural accelerators. When mapping BNNs onto the specialized neural accelerators that adopt fixed-point feature data representation and binary parameters, due to the operation overflow caused by short fixed-point coding, the BNN inference results from the deep learning frameworks on CPU/GPU will be inconsistent with those from the accelerators. This issue leads to a large deviation between the training environment and the inference implementation, and causes potential model accuracy losses when deployed on the accelerators. Therefore, we present a series of methods to contain the overflow phenomenon, and enable typical deep learning frameworks like Tensorflow to effectively train BNNs that could work with high accuracy and convergence speed on the specialized neural accelerators.

This work is oriented at the edge computing scenario that terminal deep learning accelerators use pre-trained neural network models distributed from third-party providers (e.g. from data center clouds) to process the private data instead of sending it to the cloud. In this scenario, the network model is exposed to the risk of being attacked in the unverified devices if the parameters and hyper-parameters are transmitted and processed in an unencrypted way. Our work tackles this security problem by using on-chip memory Physical Unclonable Functions (PUFs) and Processing-In-Memory (PIM). We allow the model execution only on authorized devices and protect the model from white-box attacks, black-box attacks and model tampering attacks. The proposed PUFs-and-PIM based Protection method for neural Models (P3M), can utilize unstable PUFs to protect the neural models in edge deep learning accelerators with negligible performance overhead. The experimental results show considerable performance improvement over two state-of-the-art solutions we evaluated.

As an energy-efficient hardware solution for deep neural network (DNN) inference, systolic accelerators are particularly popular in both embedded and datacenter computing scenarios. Despite their excellent performance and energy efficiency, however, systolic DNN accelerators are naturally facing a resource under-utilization problem - not all DNN models can well match the fixed processing elements (PEs) in a systolic array implementation, because typical DNN models vary significantly from applications to applications. Consequently, state-of-the-art hardware solutions are not expected to deliver the nominal (peak) performance and energy efficiency as claimed because of resource under-utilization. To deal with this dilemma, this study proposes a novel systolic DNN accelerator with a flexible computation mapping and dataflow scheme. By providing three types of parallelism and dynamically switching among them: channel-direction mapping, planar mapping, and hybrid, our accelerator offers the adaptability to match various DNN models to the fixed hardware resources, and thus, enables flexibly exploiting PE provision and data reuse for a wide range of DNN models to achieve optimal performance and energy efficiency.

The ever-growing processing ability of in-memory processing logic makes the data sharing and coherence between processors and in-memory logic play an increasingly important role in Processing-in-Memory (PIM) systems. Unfortunately, the existing state-of-the-art coarse-grained PIM coherence solutions suffer from unnecessary data movements and stalls caused by a data ping-pong issue. This work proposes CuckooPIM, a criticality-aware and less-blocking coherence mechanism, which can effectively avoid unnecessary data movements and stalls. Experiments reveal that CuckooPIM achieves 1.68x speedup on average comparing with coarse-grained PIM coherence.

Carbon Nanotube field-effect transistors (CNFET) emerge as a promising alternative to the conventional CMOS for the much higher speed and power efficiency. It is particularly suitable for building the power-hungry last level cache (LLC). However, the process variation (PV) in CNFET substantially affects the operation stability and thus the worst-case timing, which limits the LLC operation frequency dramatically given a fully synchronous design. To address this problem, we developed a variation-aware cache such that each part of the cache can run at its optimal frequency and the overall cache performance can be improved significantly.

Prior research on energy-efficient Convolutional Neural Network (CNN) inference accelerators mostly focus on exploiting the model sparsity, i.e., zero patterns in weight and activations, to reduce the on-chip storage and computation overhead. In this work, we found in addition to zero patterns, a larger group of repetitive patterns and values exists in the working-set of CNN inference task, which is defined as computation redundancy and induces unnecessary performance and storage overhead in CNN accelerators. Based on this observation, we proposed a redundancy-free architecture that detects and eliminates the repetitive computation and storage patterns in CNN for more efficient network inference. The architecture consists of two parts: the off-line parameter analyzer that extracts the repetitive patterns in the 3D tensor of parameters, and the dataflow accelerator. The proposed accelerator at first preprocesses the weight patterns and the dynamically generated activations, and then cache these intermediate results in special P2-cache banks for further usage in convolution or full-connection stage. It is evaluated in experiments that the proposed Cavoluche architecture removes up to 89% of the repetitive operations from the layer inference process and reduce 77% of on-chip storage space to store both redundancy-free weight and activations. It is seen in experiments that the implementation of Cavoluche outperforms the state-of-the-art mobile GPGPU in both performance and energy-efficiency. When compared to the latest sparsity base accelerators, Cavoluche also achieves better operation elimination effects.

3D convolutional neural networks (CNN) are gaining popularity in action/activity analysis. Compared to 2D convolutions that share the filters in 2D spatial domain, 3D convolutions further reuse filters in the temporal dimension to capture time-domain features. Prior works on specialized 3D-CNN accelerators employ additional on-chip memories and multi-cluster architecture to reuse data among the process element (PE) arrays, which is too expensive for low-power chips. Instead of harvesting in-memory locality, we propose a 3D systolic cube architecture to exploit the spatial-and-temporal localities of 3D CNNs, which moves the reusable data in-between PEs connected via a 3D-cube Network-on-Chip. Evaluation shows that systolic-cube contributes to considerable energy-efficiency boost for activity-recognition benchmarks.

Accelerating deep neural networks on resource-constrained embedded devices is becoming increasingly important for real-time applications. However, in contrast to the intensive research works on specialized neural network inference architectures, there is a lack of study on the acceleration and parallelization of deep learning inference on embedded chip-multiprocessor architectures, which are favored by many real-time applications for superb energy-efficiency and scalability. In this work, we investigate the strategies of parallelizing single-pass deep neural network inference on embedded on-chip multi-core accelerators. These methods exploit the elasticity and noise-tolerance features of deep learning algorithms to circumvent the bottleneck of on-chip inter-core data moving and reduce the communication overhead aggravated as the core number scales up. The experimental results show that the communication-aware sparsified parallelization method improves the system performance by 1.6×−1.1× and achieves 4×−1.6× better interconnects energy efficiency for different neural networks.

Compute-centric architecture is suffering from the data moving overhead caused by memory wall, particularly for applications like large-scale data analysis based on deep learning technology. In this work, we designed an energy-efficient In-SSD Deep Learning Accelerator, InS-DLA, for Near-Data-Processing. InS-DLA directly operates on NAND Flash inside the Open Channel Solid-State-Drive where the target data are stored, eliminating the power and performance overhead caused by data movement. Experimental results reveal that the InS-DLA based FPGA prototype reduces energy consumption by 95.82% and 59.23%, compared to conventional CPU and GPU based deep learning systems.

Precision-tuning is a popular approach of approximate computing to trade-off excessive computation exactness for power and efficiency gains. Particularly, it has been proved useful to reduce the computation and memory overhead for the deep neural networks on embedded and IoT usage. However, the switching overhead of precision tuning in hardware severely impacts its applicability and effectiveness to save more energy by quickly reacting to the change of environment, user constraint or input quality. This work for the first time investigates the feasibility of agile and cost-free precision tuning for neural network accelerators to benefit from approximate computing. The proposed Processing in Memory (PIM) CNN accelerators fully utilize the normally-off characteristics of memristor crossbars to achieve instant network precision tuning without worrying about the model reloading penalty. The ReRAM-based accelerator, with the proposed neural parameter mapping policy and the novel mixed-model training method, involves negligible precision-switching latency and power consumption compared with traditional variable precision accelerators. The proposed mixed-model training perfectly unifies the neural models of different precision into a single ReRAM array without compromising the accuracy, and the ReRAM accelerator could save 58.3%-62.47% area overhead compared with conventional designs that have to program multiple independent models into ReRAM arrays for precision tuning.

The technological breakthrough in Generative Adversarial Networks (GAN) has propelled the advancement of content generative applications such as AI-based paintings, style transfer, and music composition. However, in contrast to previous deep learning models for prediction and categorization, generative networks generally rely on instance normalization (IN) layer for better feature distribution, which performs significantly better than batch normalization(BN) in image style-transfer, image to image translation, etc. Unlike batch or group normalization that can be fused into convolutional layers and ignored during the network inference stage, an instance normalization layer induces intensive computation and memory access. However, prior deep learning accelerator designs for traditional Neural Network and Generative Adversarial Networks mostly focus on the acceleration of convolution and deconvolution layer but lack of support for IN operations, which could become a performance bottleneck on edge devices with insufficient computational power. To address this problem, we propose an inference accelerator for content generation (ACG-Engine) aimed to support the fundamental operations of generative networks, including convolution layers, deconvolution layers, specifically instance normalization layer. We performed a hardware-aware mathematical transformation of the IN operation for less computation complexity and memory-friendliness, so that it can be efficiently mapped to the classic 2D processing element array. Owing to the proposed optimization techniques, ACG-Engine achieves 4.56X speedup and improve power efficiency up to 29X compared to prior baseline acceleration scheme in generative network acceleration. In addition, ACG-Engine can achieve performance comparable to the classic CNN-specific accelerators with negligible power consumption and area overhead.

The emerging memristor technology is considered a promising solution to the edge-oriented deep learning and neuromorphic processor chips because it enables power-efficient Computing-in-Memory (CiM) and normally-off architecture simultaneously. However, as the analog nature and the immature nano-scale fabrication technology, the memristive cells suffer from manufacturing defects, process variations and aging-induced variations, which may incur system and function failures in applications. How to detect and rescue from the permanent and soft faults poses a significant challenge to the edge ReRAM-based deep learning or neuromorphic chips. In this work, we propose an edge-cloud collaborative framework, RRAMedy, to achieve in-situ fault detection and network remedy for memristor-based neural accelerators. In this framework, we present Adversarial Example Testing, a lifetime on-device fault detection technique, which can accurately detect defected cells and memristor soft faults with high probability and at a low cost. Furthermore, the model accuracy can be restored by the proposed edge-cloud collaborative fault-masking retraining and model updating mechanism with a minimized edge-cloud communication overhead. The experimental results show that RRAMedy can effectively detect the memristor permanent and soft faults, protecting the neural accelerator from accuracy and performance degradation in its life cycle.

Neural networks especially the convolution neural networks (CNN) have become prevalent and numerous CNN accelerators have been developed to achieve higher performance. While clock frequency determines the operation speed and has direct influence on the performance of the accelerators, we propose to apply overclocking, a circuit optimization approach that enables higher clock frequency, on general CNN accelerators. This technique brings significant performance improvement, but it leads to moderate timing errors, wrong computing results and low prediction accuracy. By taking advantage of the inherent fault tolerance of neural networks, we opt to learn the computing errors together with the application data with additional on-accelerator training. In this case, the resulting models can be resilient to the errors and do not necessarily suffer considerable prediction accuracy loss. In addition, we also take the worst case of overclocking into consideration with a series of approaches ranging from fault detection to fault recovery in case of hardware crash. Finally, we demonstrate the use of overclocking on a CNN accelerator implemented on Xilinx KCU1500 with comprehensive experiments. The experiments show that overclocking in combination with the on-accelerator neural network training improves both the neural network performance and energy efficiency with small prediction accuracy loss.

There is a sheer growth of attention on the issue of deep learning attacks launched by adversaries. Especially with the spread of edge computing devices that cooperate with the central cloud, how to protect the neural network models and private data from being attacked becomes a hot topic. In this paper, we consider two collaborative edge-cloud deep learning scenarios. The first is that deep learning models are trained on resource-rich cloud and deployed on the terminal deep learning accelerators for delay-sensitive tasks. While in the second scenario, edge collected data is offloaded to the cloud for computationally-intensive tasks. In both scenarios, the valuable pre-trained neural network models and the private data are exposed to the risks of being attacked if they are transmitted and processed in an unencrypted way. To tackle this security problem, we present a lightweight protection scheme towards data-oriented attacks and model-oriented attacks. With the use of on-chip memory Physical Unclonable Functions (PUFs) and Processing-In-Memory (PIM), our method limits the model execution only on specific edge devices and also prevents the unauthorized analysis of private data. Experiments on state-of-the-art deep learning networks show that our method secures edge deep learning models and user-data with a negligible performance overhead.

Recently the development of deep learning has been propelling the sheer growth of vision and speech applications on lightweight embedded and mobile systems. However, the limitation of computation resource and power delivery capability in embedded platforms is recognized as a significant bottleneck that prevents the systems from providing real-time deep learning ability, since the inference of deep convolutional neural networks (CNNs) and recurrent neural networks (RNNs) involves large quantities of weights and operations. Particularly, how to provide quality-of-services (QoS)-guaranteed neural network inference ability in the multitask execution environment of multicore SoCs is even more complicated due to the existence of resource contention. In this article, we present a novel deep neural network architecture, MV-Net, which provides performance elasticity and contention-aware self-scheduling ability for QoS enhancement in mobile computing systems. When the constraints of QoS, output accuracy, and resource contention status of the system change, MV-Net can dynamically reconfigure the corresponding neural network propagation paths and thus achieves an effective tradeoff between neural network computational complexity and prediction accuracy via approximate computing. The experimental results show that (1) MV-Net significantly improves the performance flexibility of current CNN models and makes it possible to provide always-guaranteed QoS in a multitask environment, and (2) it satisfies the quality-of-results (QoR) requirement, outperforming the baseline implementation significantly, and improves the system energy efficiency at the same time.

Recently powerful convolutional neural network (CNN) accelerators are emerging as energy-efficient solutions for real-time vision/speech processing, recognition and a wide spectrum of approximate computing applications. In addition to the broad applicability scope of such deep learning (DL) accelerators, we found that the fascinating feature of deterministic performance makes them ideal candidates as application-processors in embedded SoCs concerned with real-time processing. However, unlike traditional accelerator designs, DL accelerators introduce the new aspect of design tradeoff between real-time processing [quality of service (QoS)] and computation approximation [quality of result (QoR)] into embedded systems. This paper proposes an elastic CNN acceleration architecture that automatically adapts to the user-specified QoS constraint by exploiting the error-resilience in typical approximate computing workloads. For the first time, the proposed design, including the network tuning-and-mapping software and reconfigurable accelerator hardware, aims to reconcile the design constraint of QoS and QoR, which are respectively, the critical concerns in real-time and approximate computing. It is shown in experiments the proposed architecture enables the embedded system to work flexibly in an expanded operating space, significantly enhances its real-time ability, and maximizes the system energy-efficiency within the user-specified QoS-QoR constraint through self-reconfiguration. Also, we showcase the application of the proposed design approach to lower power image recognition challenge (LPIRC) and how it is employed to forge an energy-efficient solution to the LPIRC contest.

Trace signal selection is of great importance for post-silicon debug. Debuggers traditionally use state restoration to improve the observability of the trace data, and state restoration ratio (SRR) is computed after state restoration. In this paper, we exploit the combination of snapshot states and trace states to improve the observability. First, we propose a novel state restoration method, called cluster restoration. It uses both the snapshot states of flip-flop clusters at the beginning of tracing, and the tracing states of the clusters' inputs during the tracing window to deterministically restore all states of these clusters during the tracing window. We also present a cluster restoration-based trace signal selection method to select clusters instead of trace signals directly, which includes two stages: 1) cluster generation and 2) cluster evaluation. For cluster generation, feedback loop-based cluster generation and backward tracing-based cluster generation techniques are proposed. For cluster evaluation, a new metric, called the global state restoration improvement is proposed to evaluate the candidate clusters. The experimental results show that in comparison to prior trace signal selection methods, our method can improve the SRR and reduce the runtime of trace signal selection as well.

Considering the insatiable demand for high-performance computing, on-chip cache capacity increases rapidly. Spin-transfer-torque magnetoresistive random-access memory (STT-MRAM) is a promising cache candidate due to ultralow standby power, high-access speed, and integration density. Unfortunately, when the feature size of magnetic tunnel junction (MTJ) scales down to 1 Xnm, read current approaches write current closely, which may result in read disturbance threatening the reliability of STT-MRAM. Furthermore, the elevating on-chip temperature reduces the thermal stability of STT-MRAM remarkably and aggravates the read disturbance. Error correction code (ECC) is an effective technique to enhance memory reliability. In this paper, we take advantage of the thermal dependence of STT-MRAM and propose a thermally adaptive ECC design, called “Chameleon,” that can adjust the ECC protection strength dynamically to reduce the ECC storage overhead and improve the cache access performance and energy efficiency. Experimental results show that compared to the conservative nonadaptive ECC scheme, our design can improve both cache performance and energy consumption effectively.

Conventional high-performance processors employ hardware data prefetcher to reduce the cache miss rate by exploiting data locality in memory access. In the scenario of PIM-enabled system, current prefetching methods are likely to induce unnecessary data moving between memory and CPU, resulting in significant performance slow-down if ignoring the data access pattern of the In-Memory cores of PIM architecture. In this paper, we propose a novel PIM-oriented Prefetching (PIMCH), to fix this problem for the heterogeneous system with both conventional CPU cores and PIM architectures. PIMCH consider the memory behavior of both CPU cores and PIM cores, and refers to their data mapping to dynamically adjust prefetching methods, which enables cooperative memory prefetching in Processing-In-Memory architecture.

The resurrection of Processing in memory (PIM) architectures is expected to address the ever-worsening memory wall issue in big data era. In this work, we propose XORiM, an inexpensive PIM design to achieve fast bulky bitwise XOR operation in commodity DRAM devices for memory-intensive workloads. Instead of resorting to 3D-integration or emerging memory technology, we reuse and adapt the peripheral circuits and row-buffers in memory to enable within-DRAM data manipulation. The implemented mechanism can also be employed to conduct high-throughput bulky data operations including memory initialization, AND, OR and INV. We present the detailed circuitry design and transistor-level simulation to evaluate the proposed method, and demonstrate the application of XO-RiM to realistic workloads by conducting full-system level simulation. The experimental results on data-intensive applications such as deduplication and data encryption show that about 1.5× and 5.1× overall performance benefits and 4.9× and 9.1× overall energy savings are achieved respectively by XORiM over conventional computing systems.

As a critical operation in robotics, motion planning consumes lots of time and energy, especially in a dynamic environment. Through approaches based on general-purpose processors, it is hard to get a valid planning in real time. We present an accelerator to speed up collision detection, which costs over 90% of the computation time in motion planning. Via the octree-based roadmap representation, the accelerator can be reconfigured online and support large roadmaps. We in addition propose an effective algorithm to update the roadmap in a dynamic environment, together with a batched incremental processing approach to reduce the complexity of collision detection. Experimental results show that our accelerator achieves 26.5X speedup than an existing CPU-based approach. With the incremental approach, the performance further improves by 10X while the solution quality is degraded by 10% only.

The field of image and video recognition has been propelled by the rapid development of deep learning in recent years. With its fascinating accuracy and generalization ability, deep CNNs have shown remarkable performance in large-scale and real-life image dataset. However, accommodating computation-intensive CNN-based image detection frameworks on power-constrained devices is considered more challenging than desktop or warehouse computing systems. Instead of emphasizing purely on detection accuracy, Low Power Image Recognition Challenge (LPIRC) is initiated to highlight the energy-efficiency of different image recognition solutions, and it witnesses the advancement of cost-effective image recognition technology in aspects of both algorithmic and architecture innovation. This paper introduces the cost-effective CNN-based object detection solutions that reached an improved tradeoff between energy and accuracy for mobile CPU+GPU SoCs, which is the winner of LPIRC2016, and it also analyzes the implications of both recent hardware and algorithm advancement on such a technique. It is demonstrated in our evaluation that the performance growth of embedded SoCs and CNN models have clearly contributed to a sheer growth of mAP/WH in current CNN-based object detection solutions, and also shifted the balance between accuracy and energy-cost in the contest solution design when we seek to maximize the efficiency score defined by LPIRC through design parameter exploration.

Unlike standard Convolutional Neural Networks (CNNs) with fully-connected layers, Fully Convolutional Neural Networks (FCN) are prevalent in computer vision applications such as object detection, semantic/image segmentation, and the most popular generative tasks based on Generative Adversarial Networks (GAN). In an FCN, traditional convolutional layers and deconvolutional layers contribute to the majority of the computation complexity. However, prior deep learning accelerator designs mostly focus on CNN optimization. They either use independent compute-resources to handle deconvolution or convert deconvolutional layers (Deconv) into general convolution operations, which arouses considerable overhead. To address this problem, we propose a unified fully convolutional accelerator aiming to handle both the deconvolutional and convolutional layers with a single processing element (PE) array. We re-optimize the conventional CNN accelerator architecture of regular 2D processing elements array, to enable it more efficiently support the data flow of deconvolutional layer inference. By exploiting the locality in deconvolutional filters, this architecture reduces the consumption of on-chip memory communication from 24.79 GB to 6.56 GB and improves the power efficiency significantly. Compared to prior baseline deconvolution acceleration scheme, the proposed accelerator achieves 1.3X–44.9X speedup and reduces the energy consumption by 14.60/0-97.6% on a set of representative benchmark applications. Meanwhile, it keeps similar CNN inference performance to that of an optimized CNN-only accelerator with negligible power consumption and chip area overhead.

As the technology node shrinks, leakage power becomes a bottleneck for processor performance and memory capacity scalings. Spin Torque Transfer Magnetic Random Access Memory (STT-MRAM) has negligible leakage power, fast access speed, high integration density and non-volatility. Therefore, it is a promising candidate for the last level cache design. However, it suffers from high write energy and slow write speed. In the paper, we observe that the traditional cache replacement policy is not optimal when applied to STT-MRAM from the energy consumption perspective. So we propose a novel write energy aware cache replacement policy, which utilizes a MinHash function to identify the similarities between the cache line to be written back and candidates for the replacement. The cache line with the highest similarity is chosen as the victim. In addition, we propose a new metric for cache replacement considering both performance and write energy to improve the replacement policy further. The experimental results show that our proposed policy can reduce write energy by 33.6% on average compared to the state-of-the-art Least Recently Used (LRU) replacement policy with only 0.5% performance penalty and negligible hardware overhead.

Processing-in-Memory (PIM) has been recognized as the most feasible solution to resolve the ever-aggravating memory wall especially as the boom of memory-intensive scale-out workloads such as graph computing and data analytics. However, when the future computing system becomes more and more likely to adopt PIM architectures as a type of the storage and processing unit, existing aging-award task scheduling algorithms for heterogeneous systems do not consider memory interference in PIM+CPU system, deducing an inaccurate task runtime and temperature which will over-estimate MTTF. We proposed a quantitatively formalized model for the aging reliability of PIM+CPU heterogeneous system and MTTF-ALG (a MTTF-based task scheduling algorithm) to balance the MTTF of whole system. Experimental results show that, compared to the traditional scheduling algorithm for heterogeneous system, the proposed method is able to reduce MTTF variation over 60.2% on average and the runtime by 15.3% on average for PIM+CPU system.

The shared memory controllers of single-chip cloud computing (SCC) processors are vulnerable to timing-channel attacks. Existing protection strategies based on fixed memory bandwidth assignment degrade processor performance and severely harm the experience of cloud-users. This paper proposes a novel light-weight timing channel protection scheme against both side channel and covert channel attacks to the shared memory controllers of cloud chip multi-processors (CMPs). Instead of enforcing a fixed time-slot assignment to the applications, we go an entirely different technical route and utilize the background DRAM refresh as a free noise source to eliminate the time correlation between victim and attacker applications. The proposed protection framework, MemJam, relies on the emerging fine-grained refresh technology to achieve the effects of timing channel obfuscation. Multi-programmed workloads running in a cloud CMP were used to evaluate the protection method. The results show that the light-weight refresh-based noise can effectively block the timing-channel between user applications, and achieve up to 1.61~2.49X memory performance boost compared to prior solutions.

The shared memory controllers of Single-Chip Cloud computing processors are vulnerable to timing-channel attacks. Existing protection strategies based on fixed memory bandwidth assignment degrade processor performance and severely harm the experience of cloud-users. This paper proposes a novel light-weight timing channel protection scheme against both side channel and covert channel attacks to the shared memory controllers of cloud Chip Multi-Processors (CMPs). Instead of enforcing a fixed time-slot assignment to the applications, we go an entirely different technical route and utilize the background DRAM refresh as a free noise source to eliminate the time correlation between victim and attacker applications. The proposed protection framework, MemJam, relies on emerging Fine-Grained Refresh technology to achieve the effects of timing channel obfuscation. Multi-programmed workloads running in a cloud CMP were used to evaluate the protection method. The results show that the light-weight refresh-based noise can effectively block the timing-channel between user applications, and achieve up to 1.69~3X memory performance boost compared to prior solutions.

The trigger circuitry is critical for trace-based post-silicon debug, which detects specified events or event sequences to initiate or stop the tracing. In this paper, we propose a resource efficient trigger design for the post-silicon debug which integrates several different detection schemes to improve the detect ability. The design reuses the trace buffer to store the trigger set for event detection or store the transitions of the generated finite state machine for event sequence detection, which converts the trigger detection into simple read operations to the trace buffer and equality matching operations. Simulation and emulation are both used to validate the usability of the design. In comparison with the prior trigger circuits with the same trigger width, the proposed method provides much more powerful detect ability and configurability for complicated trigger conditions, and needs lower area overhead.

Data compression has been intensively studied to increase the utility of cache, network-on-chip (NoC), and main memory in energy-efficient processors. However, prior solutions to data compression often add remarkable compression and decompression delay to the critical path of memory access, which is thought as the major factor limiting its application to commodity processors. Unlike prior work that deals with memory compression or network compression separately, this paper proposes a unified on-chip distributed data compressor (DISCO), to enable near-zero-latency cache and memory block compression for chip multiprocessors adopting nonuniform cache access. DISCO integrates a multimode cache compressor into the NoC routers and overlaps the de/compression latency with the queuing delay in the network. In addition, cache block evicted to or fetched from the main memory can also be compressed or decompressed during the network queuing time in this unified DISCO compressor. With the support of congestion-awareness, it is shown in the evaluation that DISCO, which unifies the compression solution of the memory hierarchy, dramatically decreases the compression overhead of isolated techniques, and significantly boosts the efficiency of data moving and store.

The rapid development of machine learning is enabling a plenty of novel applications, such as image and speech recognition for embedded and mobile devices. However, state-of-the-art deep learning models like convolutional neural networks (CNNs) are demanding so much on-chip storage and compute resources that they cannot be smoothly handled by low-power mobile or embedded systems. In order to fit large CNN models into mobile or more cutting-edge devices for IoT or cyberphysics applications, we proposed an efficient on-chip memory architecture for CNN inference acceleration, and showed its application to in-house single-instruction multiple-data structure machine learning processor. The redesigned on-chip memory subsystem, Memsqueezer, includes an active weight buffer and data buffer set that embraces specialized compression methods to reduce the footprint of CNN parameters (weights) and activation data, respectively. Memsqueezer buffer can compress the data and weight set according to the dataflow in computation, and it also includes a built-in redundancy detection mechanism that actively scans through the working-set of CNNs to boost their inference performance by eliminating the computation redundancy in CNN models. In our experiments, it is shown that the CNN processors with Memsqueezer buffers achieve more than 2× performance improvement and reduces 85% energy consumption on average over the conventional buffer design with the same area budget.

Multi-granularity memory system provides multiple access granularities for the applications with various spatial localities. In the multi-granularity access pattern, the one-size-bandwidth NoC design cannot utilize the bandwidth efficiently. We propose a novel NoC design, called BoDNoC, which can merge multiple narrow subnets to provide various bandwidths for access data. The new design also adopts an optimization algorithm to take full advantage of bandwidth provision. Experimental results show that BoDNoC can improve the throughput by 23.5% and reduce the energy consumption by 37.2% in comparison with one-size-bandwidth NoC design.

Processing-in-Memory (PIM), has recently been revisited as one of the most promising solutions to deal with the issue of bandwidth and power wall between processor and memory. In this paper, we propose a light-weight PIM architecture, approxPIM, which leverages approximate computing techniques to enable InMemory Processing in a realistic 3D-stacked DRAM, Micron's Hybrid Memory Cube (HMC). Using the newly-released atomic instruction support of the HMC, approxPIM can process a wide range of data-intensive applications without adding any logic resources into the memory devices. Furthermore, we propose to approximate those accuracy-insensitive applications with the limited functioning set of HMC commands so that they could be smoothly mapped to the HMCs without the inference from processors, therefore enabling energy-efficient Processing-in-Memory and greatly expanding the scope of target PIM applications with HMC. In general, approxPIM gives a comprehensive study on HMC's potential and weakness in the application of Processing-in-Memory. Evaluation results show that our approxPIM significantly boosts the energy-efficiency and performance of the whole system.

This paper surveys how to use Convolutional Neural Networks (CNN) to hypothesize object location and categorization from images or videos in mobile heterogeneous SoCs. Recently a variety of CNN-based object detection frameworks have demonstrated both increasing accuracy and speed. Though they are making fast progress in high quality image recognition, state-of-the-art CNN-based detection frameworks seldom discuss their hardware-depended aspects and the cost-effectiveness of real-time image analysis in off-the-shelf low-power devices. As the focus of deep learning and convolutional neural nets is shifting to the embedded or mobile applications with limited power and computational resources, scaling down object detection framework and CNNs is becoming a new and important direction. In this work we conduct a comprehensive comparative study of state-of-the-art real-time object detection frameworks about their performance, cost-effectiveness/energy-efficiency (in the metric of mAP/Wh) in off-the-shelf mobile GPU devices. Based on the analysis results and observation in investigation, we propose to adjust the design parameters of such frameworks and employ a design space exploration procedure to maximize the energy-efficiency (mAP/Wh) of real-time object detection solution in mobile GPUs. As shown in the benchmarking result, we successfully boost the energy-efficiency of multiple popular CNN-based detection solutions by maximizing the utility of computation resources of SoC and trading-off between prediction accuracy and energy cost. In the second Low-Power Image Recognition Challenge (LPIRC), our system achieved the best result measured in mAP/Energy on the embedded Jetson TX1 CPU+GPU SoC.

The search for alternative memory technologies has attracted significant attention toward emerging non-volatile memories. Among them, STT-MRAM, PCM, RRAM have shown promising characteristic to gain a position inside the memory hierarchy of computing platforms, and even enable new computing paradigms. However like any other emerging technology these devices are affected by concerns to be resolved before they could become a mainstream. This paper reviews the main reliability and testability challenges of aforementioned emerging non-volatile memories and highlights the main future considerations toward them.

Kinematics is the basis of robotic control, which manages the robots' movement, walking and balancing. As a critical part of Kinematics, the Inverse Kinematics (IK) will consume more time and energy to figure out the solution with the degrees of freedom increase. It goes beyond the ability of general-purpose processor based methods to provide real-time IK solver for manipulators with high degree of freedom. In this paper, we present a novel parallel algorithm, Quick-IK, based on the Jacobian transpose method. Via speculative searching in parallel, Quick-IK can reduce the number of iterations by 97% for the baseline Jacobian transpose method. In addition, we propose a novel specialized architecture, IKAcc, to boost the energy efficiency of Quick-IK through hardware acceleration. The evaluation shows that IKAcc can solve IK problem in 12 milliseconds for a 100 degrees of freedom manipulator. In addition, IKAcc can achieve 1700x performance speed-up over the CPU implementation of the original Jacobian transpose method and 30x speedup over the GPU implementation of Quick-IK. At same time, IKAcc achieves about 776x higher energy efficiency than the GPU implementation of Quick-IK.

Due to the recent progress in deep learning and neural acceleration architectures, specialized deep neural network or convolutional neural network (CNNs) accelerators are expected to provide an energy-efficient solution for real-time vision/speech processing. recognition and a wide spectrum of approximate computing applications. In addition to their wide applicability scope, we also found that the fascinating feature of deterministic performance and high energy-efficiency, makes such deep learning (DL) accelerators ideal candidates as application-processor IPs in embedded SoCs concerned with real-time processing. However, unlike traditional accelerator designs, DL accelerators introduce a new aspect of design trade-off between real-time processing (QoS) and computation approximation (QoR) into embedded systems. This work proposes an elastic CNN acceleration architecture that automatically adapts to the hard QoS constraint by exploiting the error-resilience in typical approximate computing workloads For the first time, the proposed design, including network tuning-and-mapping software and reconfigurable accelerator hardware, aims to reconcile the design constraint of QoS and Quality of Result (QoR). which are respectively the key concerns in real-time and approximate computing. It is shown in experiments that the proposed architecture enables the embedded system to work flexibly in an expanded operating space, significantly enhances its real-time ability. and maximizes the energy-efficiency of system within the user-specified QoS-QoR constraint through self-reconfiguration.

Due to the increasing complexity of the processor architecture and the time-consuming software simulation, efficient design space exploration (DSE) has become a critical challenge in processor design. To address this challenge, recently machine learning techniques have been widely explored for predicting the performance of various configurations through conducting only a small number of simulations as the training samples. However, most existing methods randomly select some samples for simulation from the entire configuration space as training samples to build program-specific predictors. When a new program is considered, a large number of new program-specific simulations are needed for building a new predictor. Thus considerable simulation cost is required for each program. In this paper, we propose an efficient cross-program DSE framework TrEE by combining a flexible statistical sampling strategy and ensemble transfer learning technique. Specifically, TrEE includes the following two phases which also form our major contributions: 1) proposing an orthogonal array based foldover design for flexibly sampling the representative configurations for simulation, and 2) proposing an ensemble transfer learning algorithm that can effectively transfer knowledge among different types of programs for improving the prediction performance for the new program. We evaluate the proposed TrEE on the benchmarks of SPEC CPU 2006 suite. The results demonstrate that TrEE is much more efficient and robust than state-of-art DSE techniques.

As the speed gap of the modern processor and the off-chip main memory enlarges, on-chip cache capacity increases to sustain the performance scaling. As a result, the cache power occupies a large portion of the total power budget. STT-MRAM (Spin Transfer Torque Magnetic Memory) is proposed as a promising solution for the low power cache design due to its high integration density and ultra-low leakage. Nevertheless, the high write power and latency of STT-MRAM become new barriers for the commercialization of this emerging technology. In this paper, we investigate the thermal effect on the access performance of STT-MRAM and observe that the temperature can affect the write delay and energy significantly. Then, we explore the NUCA (Non-Uniform Cache Access) design of the CMPs (Chip-Multi-Processors)with STT-MRAM based LLC (Last Level Cache). A thermal aware data migration policy, called “Thermosiphon”, which takes advantage of the thermal property of STT-MRAM, is proposed to reduce the LLC write energy. This policy splits the LLC into different regions based on the thermal distribution and adaptively migrate write intensive data considering the temperature gradient among different thermal regions. Compared to the conventional NUCA design, our proposed design can save 22.5% write energy with negligible hardware overhead.

Efficient execution of distributed database operators such as joining and aggregating is critical for the performance of big data analytics. With the increase of the compute speedup of modern CPUs, reducing the network communication time of these operators in large systems is becoming increasingly important, and also challenging current techniques. Significant performance improvements have been achieved by using state-of-the-art methods, such as reducing network traffic designed in the data management domain, and data flow scheduling in the data communications domain. However, the proposed techniques in both fields just view each other as a black box, and performance gains from a co-optimization perspective have not yet been explored. In this paper, based on current research in coflow scheduling, we propose a novel Coflow-based Co-optimization Framework (CCF), which can co-optimize application-level data movement and network-level data communications for distributed operators, and consequently contribute to their performance in large distributed environments. We present the detailed design and implementation of CCF, and conduct an experimental evaluation of CCF using large-scale simulations on large data joins. Our results demonstrate that CCF can always perform faster than current approaches on network communications in large-scale distributed scenarios.

The lack of observability of prototype chips makes post silicon debug extremely difficult and time consuming. Trace based debug techniques can improve the observability by acquiring some internal states at runtime through a dedicated on-chip trace buffer. In this paper, we propose a flip-flop clustering based trace signal selection method, which uses the forward tracing to generate flip-flop clusters and evaluates the global state restoration improvement to select the clusters for state capturing. The predecessor flip-flops of the selected cluster are used as trace signals, and the internal flip-flops of the selected clusters are used as snapshot signals. The experimental results show that in comparison with the prior methods, our method can increase the state restoration ratio by 74.8% on average with no more than 1.58% storage overhead for storing snapshots. The runtime for trace signal selection is also much less than those of prior methods.

Phase change memory (PCM) is a promising alternative to Dynamic Random Access Memory (DRAM) as main memory due to its merits of high density and low leakage power. Multi-level Cell (MLC) PCM is more attractive than Single-level Cell (SLC) PCM, because it can store multiple bits per cell to achieve higher density and lower per-bit cost. With the iterative program-verify write technique, MLC PCM writes demand at much higher power than DRAM writes, while the power supply system of MLC memory system is similar to that of DRAM, and the power capability is limited. The incompatibility of high write power and limited power budget results in the degradation of the write throughput and performance in MLC PCM. In this work, we investigate both write scheduling policy and power management to improve the MLC power utility and alleviate the negative impacts induced by high write power. We identify the power-utility-driven write scheduling as an online bin-packing problem and then derive a power-utility-driven scheduling (PUDS) policy from the First Fit algorithm to improve the write power usage. Based on the ramp-down characteristic of the SET pulse (the pulse changes the PCM to high resistance), we propose the SET Power Amortization (SPA) policy, which proactively reclaims the power tokens at the intra-SET level to promote the power utilization. Our experimental results demonstrate that the PUDS and SPA respectively achieve 24% and 27% performance improvement over the state-of-the-art power management technique, and the PUDS8SPA has an overall 31% improvement of the power utility and 50% increase of performance compared to the baseline system.

Refresh operations consume substantial energy and bandwidth in high-density dynamic random-access memory (DRAM) memory. The trend of increasing refresh overhead limits the scalability of DRAM memory that refreshes all cells at the same rate, because the refresh rate setting depends on the worst-case weak cell manufactured in unstable process technology. To cope with this issue, fine-grained refresh (FGR) is proposed to eliminate the unnecessary refresh operations caused by minor weak cells. Even JEDEC's DDR4 DRAM specification announces the support of FGR, which is likely to evolve and become a standard in future DRAM. Unfortunately, according to our key observation, the effectiveness of FGR is greatly confined by the procedure of refresh-oblivious device integration because all memory devices within a module have to be controlled and refreshed in a lockstep way after the step of assembly. In this paper, we are the first to propose a holistic FGR-oriented DRAM optimization framework, retention-aware DRAM assembly and repair (RADAR), to enhance the effectiveness of FGR in DRAM modules. RADAR includes two novel techniques applicable at the stage of DRAM assembly. The first one is retention-aware device clustering that integrates the “compatible” devices to achieve low refresh rate through a preassembly testing and retention-aware matching method. The second technique, Microfix, exploits the hierarchical DRAM array structure and its redundancy to fix critical weak DRAM rows through fine-grained row and subarray remapping. With this optimization architecture, RADAR, the refresh overhead of DRAM dual in-line memory modules can be dramatically reduced as implied in the experiments.

Multilevel spin toque transfer RAM (STT-RAM) is a suitable storage device for energy-efficient neural network accelerators (NNAs), which relies on large-capacity on-chip memory to support brain-inspired large-scale learning models from conventional artificial neural networks to current popular deep convolutional neural networks. In this paper, we investigate the application of multilevel STT-RAM to general-purpose NNAs. First, the error-resilience feature of neural networks is leveraged to tolerate the read/write reliability issue in multilevel cell STT-RAM using approximate computing. The induced read/write failures at the expense of higher storage density can be effectively masked by a wide spectrum of NN applications with intrinsic forgiveness. Second, we present a precision-tunable STT-RAM buffer for the popular general-purpose NNA. The targeted STT-RAM memory design is able to transform between multiple working modes and adaptable to meet the varying quality constraint of approximate applications. Lastly, the reconfigurable STT-RAM buffer not only enables precision scaling in NNA but also provides adaptiveness to the demand for different learning models with distinct working-set sizes. Particularly, we demonstrate the concept of capacity/precision-tunable STT-RAM memory with the emerging reconfigurable deep NNA and elaborate on the data mapping and storage mode switching policy in STT-RAM memory to achieve the best energy efficiency of approximate computing.

Unlike conventional ICs, approximate computing chips are less sensitive to hardware errors. This fascinating feature can be utilized to improve the performance of chip design and even change the timing closure procedure of digital circuit design flow. In this paper, we study the potential of resilience-aware circuit clocking scheme, and demonstrate the methodology with advanced neural network (NN)-based accelerator. We propose a novel timing analysis and frequency setting method for NN-based approximate computing circuits based on in-field NN retraining. With the proposed iterative retiming-and-retraining framework, NN-based accelerator can be retrained to operate safely at aggressive operating frequencies compared with the frequency decided purely by statistical timing analysis or Monto Carlo analysis. For nanometer process technology with increasing threats of timing errors induced by process variation, noises, and so on, our retiming-and-retraining method enables higher circuit operating frequency and enables dynamic precision/frequency adjustment for approximate computing circuits. We evaluate the methodology with both the neural and deep learning accelerators in experiments. The experimental results show that timing errors in neural circuits can be effectively tamed for different applications, so that the circuits can operate at higher clocking rates under the specified quality constraint or be dynamically scaled to work at a wide range of frequency states with only minor accuracy losses.

Convolutional neural networks (CNN) accelerators have been proposed as an efficient hardware solution for deep learning based applications, which are known to be both compute-and-memory intensive. Although the most advanced CNN accelerators can deliver high computational throughput, the performance is highly unstable. Once changed to accommodate a new network with different parameters like layers and kernel size, the fixed hardware structure, may no longer well match the data flows. Consequently, the accelerator will fail to deliver high performance due to the underutilization of either logic resource or memory bandwidth. To overcome this problem, we proposed a novel deep learning accelerator, which offers multiple types of data-level parallelism: inter-kernel, intra-kernel and hybrid. Our design can adaptively switch among the three types of parallelism and the corresponding data tiling schemes to dynamically match different networks or even different layers of a single network. No matter how we change the hardware configurations or network types, the proposed network mapping strategy ensures the optimal performance and energy-efficiency. Compared with previous state-of-the-art NN accelerators, it is possible to achieve a speedup of 4.0x-8.3x for some layers of the well-known large scale CNNs. For the whole phase of network forward-propagation, our design achieves 28.04% PE energy saving, 90.3% on-chip memory energy saving on average.

Data compression has been proposed to increase the utility of on-chip memory space or Network-on-Chip (NoC) bandwidth in energy-efficient processors. However, such techniques usually add additional compression and decompression latency to the critical path of memory access, which is one of the major factors limiting their application to processors. In contrast to prior work that deals with either cache compression or network compression separately, this study proposes a unified on-chip DIStributed data COmpressor, DISCO, to enable near-zero latency cache/NoC compression for chip multi-processors (CMPs) adopting Non-Uniform Cache Access (NUCA). DISCO integrates data compressors into NoC routers and seeks opportunity to overlap the de/compression latency with the NoC queuing delay through a coordinated NoC scheduling and cache compression mechanism With the support of DISCO that unifies the solutions of on-chip data compression, it is shown in evaluation that DISCO significantly boosts the efficiency of on-chip data caching and data moving.

Recent advances in Neural Networks (NN) are enabling more and more innovative applications. As an energy-efficient hardware solution, machine learning accelerators for CNNs or traditional ANNs are also gaining popularity in the area of embedded vision, robotics and cyberphysics. However, the design parameters of NN models vary significantly from application to application. Hence, it's hard to provide one general and highly-efficient hardware solution to accommodate all of them, and it is also impractical for the domain-specific developers to customize their flown hardware targeting on a specific NN model. To deal with this dilemma, this study proposes a design automation tool, DeepBurning, allowing the application developers to build from scratch learning accelerators that targets their specific NN models with custom configurations and optimized performance. DeepBurning includes a RTL-level accelerator generator and a coordinated compiler that generates the control flow and data layout under the user-specified constraints. The results can be used to implement FPGA-based NN accelerator or help generate chip design for early design stage. In general, DeepBurning supports a large family of NN models, and greatly simplifies the design flow of NN accelerators for the machine learning or AI application developers. The evaluation shows that the generated learning accelerators burnt to our FPGA board exhibit great power efficiency compared to state-of-the-art FPGA-based solutions.

The rapid development of deep learning are enabling a plenty of novel applications such as image and speech recognition for embedded systems, robotics or smart wearable devices. However, typical deep learning models like deep convolutional neural networks (CNNs) consume so much on-chip storage and high-throughput compute resources that they cannot be easily handled by mobile or embedded devices with thrifty silicon and power budget. In order to enable large CNN models in mobile or more cutting-edge devices for IoT or cyberphysics applications, we proposed an efficient on-chip memory architecture for CNN inference acceleration, and showed its application to our in-house general-purpose deep learning accelerator. The redesigned on-chip memory subsystem, Memsqueezer, includes an active weight buffer set and data buffer set that embrace specialized compression methods to reduce the footprint of CNN weight and data set respectively. The Memsqueezer buffer can compress the data and weight set according to their distinct features, and it also includes a built-in redundancy detection mechanism that actively scans through the work-set of CNNs to boost their inference performance by eliminating the data redundancy. In our experiment, it is shown that the CNN accelerators with Memsqueezer buffers achieves more than 2× performance improvement and reduces 80% energy consumption on average over the conventional buffer design with the same area budget.

The limited write endurance is one of the major obstacles for phase-change random access memory (PRAM)-based main memory. Traditionally, wear-leveling (WL) techniques were proposed to enhance its lifetime by balancing write traffic. However, these techniques do not concern the endurance variation in PRAM chips. When different PRAM cells have distinct endurance, balanced writes results in lifetime degradation due to the weakest cells. In this paper, we first define a new metric-wear rate (i.e., writes/endurance) considering both the write traffic and endurance distribution from application and hardware, respectively. After investigating the writing behavior of applications and endurance variation, we propose an architecture-level leveling mechanism to balance wear rate of cells across the PRAM chip. Hardware and algorithm to support the proposed leveling mechanism are presented. Moreover, there is an important tradeoff between endurance improvement and swapping data volume. To co-optimize endurance and swapping, this situation is formulated as a maximum weight perfect matching problem in bipartite graph. Thereafter, a novel algorithm that minimizes wear-rate and swapping by employing Kuhn-Munkras algorithm is proposed to maximize PRAM lifetime and minimize performance degradation. The experimental results show ~17× lifetime improvement over prior WL.

In this paper, we investigate the feasibility of voltage adjustment in a large capacity cache, and propose the architecture of voltage-adaptable nonuniform cache access (VANUCA) that exploits near-threshold computing and multivoltage domain to approach the limit of Vddin a low-power cache. However, the adoption of near-threshold voltage (NTV) leads to a rocketing error probability in SRAM arrays, which has to be addressed by effective fault-tolerant techniques. Instead of using error correction code or data duplication, the VANUCA exploits the natural data redundancy across the whole memory hierarchy to enable fast fault recovery in the NTV cache. Based on the discovered data resilience and the multi-Vddarchitecture, the VANUCA is able to match vulnerable/invulnerable data clusters to available high-/low-voltage domains by utilizing the data migration mechanism in dynamic NUCA. The proposed VANUCA includes two important architectural techniques: 1) static assignment that assumes a fixed voltage domain partitioning and 2) DataMotion that dynamically fits the working set into heterogeneous cache banks through Vddswitching. Experimental results show that the VANUCA achieves considerable improvements in energy efficiency over the conventional single-voltage domain NUCA cache.

In 3-D-stacked memory chips, the problem of power supply integrity (PSI) is aggravating due to the additional through-silicon-via resistance and the higher current density in 3-D power delivery network. In particular, for the 3-D phase-change memory (PCM) well known for its high-amplitude programming current, IR-drop violation poses a serious threat that enforces a strict guard band of requesting concurrence, and consequently reduces the write throughput. This paper presents the implication of an IR-drop phenomenon in a 3-D PCM cube, and investigates IR-drop's impacts on write management in the PCM. From the obtained SPICE simulation results, we find that the issued writes have to meet the IR-drop constraint to be reliably processed, and then propose a PSI conscious write scheduler to improve the write performance within the constraint of the IR-drops and the power budget in the 3-D PCM cube. First, a Bloom-filter-based method is proposed to avoid the invalid write decisions for the PCM. Second, to support fine-grained write management in the cutting-edge PCM, we develop an inexpensive approach, weighted token assignment (WTA), to filter out PSI-unsafe write decisions by employing a support vector machine-based learning model. Last, a write reordering policy is proposed to cooperate with WTA and optimize the total write throughput for better memory performance. In the simulated hybrid main memory composed of both dynamic random access memory and 3-D PCM, the proposed scheduler significantly improves the write throughput.

Networks-on-Chip (NoC) gradually becomes a main contributor of chip-level power consumption. Due to the temporal and spatial heterogeneity of on-chip traffic, existing power management approaches cannot adapt the NoC power consumption to its traffic intensity, and hence lead to a suboptimal power efficiency. They either resort to over-provisioned NoC design that only suits for traffic spatial distribution, or coarse-grained power gating that only serves traffic temporal variation. In this paper, we propose a novel NoC architecture called Shuttle Networks-on-Chip (ShuttleNoC). By permitting packets shuttling between multiple subnetworks, localized power adaptation can be achieved. Experimental results show that ShuttleNoC could achieve optimal power efficiency with up to 23.5% power savings and 22.3% performance boost in comparison with traditional heterogeneity-agnostic NoC designs.

With the growth of cloud computing, security and privacy is becoming more and more important. Timing channel attack is one of the most remarkable security threads for memory controllers due to competition for shared resources. However, the existing protection strategies that ensure the deterministic of memory accesses by dividing bandwidth introduce great latency and performance degradation. This paper proposes a refresh hiding approach that adjusts the refresh operations to multiplex refresh time with additional latency introduced by those bandwidth division strategies. The experiment results show refresh hiding can reduce more than 20% of program runtime, and it will be more efficient as DRAM density increases.

Network-on-chip (NoC) is a major communication technique for 3D integrated circuits (ICs). In order to achieve higher throughput and lower latency with less system cost, horizontal and vertical wireless links are adopted to apply in the 3D NoCs. So far, the reliable routing scheme has been regarded as a lightweight and high-efficiency mechanism to guarantee the performance of the faulty 2D/3D NoCs. In this paper, we propose a low-overhead turn-guided reliable routing scheme named TWiN for the vertical link faults in wireless 3D NoCs. TWiN is deadlock-free without any virtual channels (VCs). Experimental results show that TWiN possesses higher performance, improved reliability and lower overhead compared with the state-of-the-art reliable routing scheme for wireless 3D NoCs.

Emerging highly-parallel and big data applications have renewed the research interest in Processing-in-Memory (PIM) architectures. However, moving powerful processing unit into the CMOS-incompatible DRAM chips is not cost-effective for large capacity memory. In this work, we observe that Non-Volatile Memory is often naturally incorporated with basic logics like Data Comparison Write or Flip-n-Write modules that are essential for cell SET/REST operation. In contrast to other conventional PIM or Near Data Computing (NDC) architectures, ProPRAM, as a typical Active NVM, abandons the design approach of moving accelerators or customized processors into memory devices, but begins with exploiting the existing resources inside the memory chips to accelerate the key non-compute-intensive functions for emerging big data applications. With slight hardware and architectural modification, we succeed to expose the transparent peripheral logics to the application layer through instruction set extension and exploit them for in-field bulk data processing with limited hardware cost. Compared to conventional CPU-centric systems, ProPRAM achieves an excellent optimization on energy-efficiency (15x) for important data-intensive micro-benchmarks and kernels.

Refresh operations consume substantial energy and bandwidth in high-density DRAM memory. To cope with this issue, Fine-Grained Refresh (FGR) is recently proposed to eliminate unnecessary refresh operations caused by minor weak cells. Even JEDEC's DDR4 DRAM specification announces the support of FGR to make DRAM refresh more scalable. Unfortunately, we observe that the effectiveness of FGR is greatly confined by the procedure of refresh-oblivious device integration because all memory devices within a module have to be controlled and refreshed in a lockstep way after the step of assembly. In this work, we firstly propose to intelligently integrate the "compatible" devices through a pre-assembly testing and retention-aware matching method. Second, we reuse the reconfiguration structure from yield-oriented remapping mechanism in memory chips and propose Microfix to create a balanced distribution of retention time in memory banks through fine-grained row-address tuning. With this optimization architecture, RADAR, we can eliminate the refresh overhead of produced memory modules by 28% on average.

Multi-level STT-RAM cell is able to boost the memory density at the expense of read/write reliability. However, the induced data integrity issue in STT-RAM memory can be effectively masked by a wide spectrum of applications with intrinsic forgiveness, which belong to the specific domain such as multimedia, synthesis and mining. In this work, we leverage the reconfigurable capability of MLC STT-RAM to provide variable-precision data storage for popular machine learning architectures. The targeted STT-RAM memory design is able to transform between multiple work modes and adaptable to meet the varying quality constraint of approximate applications. Particularly, we demonstrate the concept of precision-tunable STT-RAM memory with the emerging Convolution Neural Network accelerators and elaborate on the data mapping policy in STT-RAM memory to achieve the best energy-efficiency.

Workload consolidation is widely used in modern cloud processors to reduce total cost of ownership. Performance isolation has to be enforced between consolidated workloads to achieve controllable quality of service. Networks-on-chip (NoCs), as a major shared resource, often incur traffic interference and violate performance isolation criteria. Previous work resorts to strict isolation strategy that partitions NoC into independent regions to isolate core-to-core communication traffic. However, strict isolation either results in low consolidation density or degrades network performance, and more importantly, cannot be applied to memory access traffic. To address these weaknesses, we propose a novel performance isolation strategy in NoC, called relaxed isolation (RISO). It permits underutilized routers and links to be shared by multiple applications, and, at the same time, it keeps the aggregated traffic in check to enforce performance isolation. Experimental results show that RISO could effectively improve consolidation density and network performance in synergy.

The confluence of 3-D integration and network-on-chip (NoC) provides an effective solution to the scalability problem of on-chip interconnects. In 3-D integration, through-silicon via (TSV) is considered to be the most promising bonding technology. However, TSVs are also precious link resources because they consume significant chip area and possibly lead to routing congestion in the physical design stage. In addition, TSVs suffer from serious yield losses that shrink the effective TSV density. Thus, it is necessary to implement a TSV-economical 3-D NoC architecture in cost-effective design. For symmetric 3-D mesh NoCs, we observe that the TSVs bandwidth utilization is low and they rarely become the contention spots in networks as planar links. Based on this observation, we propose the TSV sharing (TS) scheme to save TSVs in 3-D NoC by enabling neighboring routers to share the vertical channels in a time division multiplexing way. We also investigate different TS implementation alternatives and show how TS improves TSV-effectiveness (TE) in multicore processors through a design space exploration. In experiments, we comprehensively evaluate TSs influence on all layers of system. It is shown that the proposed method significantly promotes TE with negligible performance overhead.

In chip multiprocessors (CMPs), nonuniform cache architecture (NUCA) is often employed to organize last-level cache (LLC) banks through network-on-chip (NoC). Because of the shrinking feature size and unstable operating environment, severe reliability problems unavoidably emerge and cause frequent on-chip component (e.g., cores, cache banks, routers) failures. Typical fault-tolerant CMPs should possess the feature of graceful degradation and function normally with deactivated tiles. However, for CMPs adopting static NUCA, certain physical address areas will become inaccessible when cache banks in a CMP node are isolated from the system. To protect cache from such threats induced by either online or offline faults, we survey several potential solutions and propose the utility-driven node remapping technique by reusing the resources in NoC. In our NoC-assisted remapping scheme, cache accesses to isolated banks are so redirected that cache space contention are successfully balanced and relieved in shared-LLC, thus ensuring the least performance penalty caused by fault isolation. Our experimental results show significant performance improvement over conventional resizing approaches such as set reduction.

With the increase of storage density, DRAM refresh leads to higher overhead of power and bandwidth, particularly in emerging 3D stacked memory design like Hybrid Memory Cube (HMC). To exploit the hardware resources for a smarter solution, we propose a data-aware refresh control scheme, Trial and Error (Trial-n-Error), which leverages the data-pattern dependence characteristics of the cells' retention time to reduce refresh operations. Trial-n-Error is a systematic approach that employs our proposed Synergy Testing to capture the refresh bottleneck of DRAM memory: “weak” cells that have a relatively shorter retention time. By locating the dominant weak cells sensitized by applications, Trial-n-Error can avoid the worst-case refresh setting, and adjust the refresh rate under the control of our self-tuning algorithm. Thus, Trial-n-Error can gradually approach to the possible lower-bound of refresh rate for less energy and memory bandwidth consumption. In experiments of 3D-stacked DRAMs, we successfully eliminate an average of 28% refresh operations and save 21% refresh energy for a set of pre-profiled synthetic data patterns and real benchmarks.

Power constraint for modern processors becomes a very serious problem with the increasing core counts and cache capacity in multi/many core processors. Compared with processing cores with mature techniques like DVFS to alleviate the situation, last level cache which consume largest portion of processor chip needs effective power management strategy. In this paper, we explore the feasibility of Near-Threshold Voltage(NTV) SRAM and Multi-Voltage Domain (Multi-VDD) for power reduction in large capacity cache. To prevent data corruption in cache, we propose redundancy-based data salvaging technique for fault recovery. To solve the dilemma of power reduction and reliability guarantee, we try to match vulnerable/invulnerable data sets to high/low voltage domains. Different from previous work, we take into consideration multi-bit errors and redundancy masking effects in Multi-VDD cache. Experimental results show that our Multi-VDD cache achieves considerable improvements in energy efficiency.

The limited write endurance of phase change random access memory (PRAM) is one of the major obstacles for PRAM-based main memory. Wear leveling techniques were proposed to extend its lifetime by balancing writes traffic. Another important concern that need to be considered is endurance variation in PRAM chips. When different PRAM cells have distinct endurance, balanced writes will result in lifetime degradation due to the weakest cells. Instead of balancing writes traffic, in this paper we propose wear rate leveling (WRL), a variant of wear leveling, to balance wear rates (i.e., writes traffic/edudrance) of cells across the PRAM chip. After investigating writing behavior of applications and endurance variation, we propose an architecture-level WRL mechanism. Moreover, there is an important tradeoff between endurance improvement and swapping data volume. To co-optimize endurance and swapping, a novel algorithm, Max Hyper-weight Rematching, is proposed to maximize PRAM lifetime and minimize performance degradation. Experimental results show 19x endurance improvement to prior Wear Leveling.

The emerging nanophotonic technology can avoid the limitation of I/O pin count, and provide abundant memory bandwidth. However, current DRAM organization has mainly been optimized for a higher storage capacity and package pin utilization. The resulted data fetching mechanism is quite inefficient in performance and energy saving, and cannot effectively utilize the abundant optical bandwidth in off-chip communication. This paper inspects the opportunity brought by optical communication, and revisits the DRAM memory architecture considering the technology trend towards multiprocessors. In our FlexMemory design, super-line prefetching is proposed to boost system performance and promote energy efficiency, which leverages the abundant photonic bandwidth to enlarge the effective data fetch size per memory cycle. To further preserve locality and maintain service parallelism for different workloads, page folding technique is employed to achieve adaptive data mapping in photonics-connected DRAM chips via optical wavelengths allocation. By combining both techniques, surplus off-chip bandwidth can be utilized and effectively managed adapting to the workloads intensity. Experimental results show that our FlexMemory achieves considerable improvements in performance and energy efficiency.

MapReduce scheduling is becoming a hot topic as MapReduce attracts more and more attention from both industry and academia. In this paper, we focus on the scheduling of mixed real-time and non-real-time applications in MapReduce environment, which is a challenging problem but receives only limited attention. To solve this problem, we present a two-level MapReduce scheduler built on previous techniques and make two key contributions. First, to meet the performance goal of real-time applications, we propose a deadline scheduler which adopts (1) a sampling based approach-Tasks Forward Scheduling (TFS) to predict map/reduce task execution time(unlike prior work that requires users to input an estimated value). (2) a resource allocation model-Approximately Uniform Minimum Degree of parallelism (AUMD) to dynamically control each realtime job to execute with minimum tasks assignment in any time so as to maximize the number of concurrent real-time jobs. Second, through integrating this deadline scheduler into existing MapReduce scheduler, we develop a two-level scheduler with resource preemption supported, and it could schedule mixed real-time and non-real-time jobs according to their respective performance demands. We implement our scheduler in Hadoop system and experiments running on a real, small-scale cluster demonstrate that it could schedule mixed real-time and nonreal-time jobs to meet their different quality-of-service (QoS) demands.

Large scale Chip-Multiprocessors (CMPs) generally employ Network-on-Chip (NoC) to connect the last level cache (LLC), which is generally organized as distributed NUCA (non-uniform cache access) arrays for scalability and efficiency. On the other hand, aggressive technology scaling induces severe reliability problems, causing on-chip components (e.g., cores, cache banks, routers) failure due to manufacture defects or on-line hardware faults. Typical degradable CMPs should possess the ability to work around defects by disabling faulty components. For static NUCA architecture, when cache banks attached to a computing node are disabled, however, certain physical address sections will no longer be accessible. Prior approaches such as sets reduction introduced in Intel Xeon processor 7100 series enable turning off cache banks by masking certain sets bits in physical address1, which greatly wastes cache capacity. In this paper, we propose to tackle the above problem in a finer granularity to restrict the capacity loss in NUCA cache. Cache accesses to isolated nodes are redirected based on the utility-driven address remapping scheme that reduces data blocks conflicts in fault-tolerant shared-LLC. We evaluate our technique using GEMS simulator. Experimental results show that address remapping achieves significant improvement over the conventional cache sizing scheme.