Implicit neural representations (INRs) have revolutionized signal encoding by using neural networks to map coordinates to signal attributes. Despite their success, INRs present significant hardware implementation challenges due to complex activation functions and floating-point operations. Unlike previous efforts, such as model pruning or quantization, we address these challenges by introducing AIRFSC, a novel activationfree stochastic computing (SC) architecture that leverages finitestate machines (FSMs). AIRFSC eliminates complex activation functions and processes data efficiently through stochastic bitstreams. Our approach decomposes the input signal into a series of Fourier basis functions, enabling the FSM-based architecture to learn smooth coordinate-to-attribute mappings for accurate signal reconstruction. Extensive experiments on diverse signal types demonstrate that AIRFSC achieves reconstruction quality comparable to state-of-the-art (SOTA) INRs implemented with multi-layer perceptrons (MLPs), while significantly improving hardware efficiency. Specifically, AIRFSC reduces power and area by $\mathbf{9 6. 8 \%}$ and $\mathbf{7 2. 8 \%}$ compared to Sinusoidal Representation Networks (SIREN), and by 97.9% and 81.3% compared to Wavelet Implicit Representation (WIRE).

Embodied Artificial Intelligence (AI) has recently attracted significant attention as it bridges AI with the physical world. Modern embodied AI systems often combine a Large Language Model (LLM)-based planner for high-level task planning and a reinforcement learning (RL)-based controller for low-level action generation, enabling embodied agents to tackle complex tasks in real-world environments. However, deploying embodied agents remains challenging due to their high computation requirements, especially for battery-powered local devices. Although techniques like lowering operating voltage can improve energy efficiency, they can introduce bit errors and result in task failures. In this work, we propose CREATE, a general design principle that leverages heterogeneous resilience at different layers for synergistic energy-reliability co-optimization. For the first time, we conduct a comprehensive error injection study on modern embodied AI systems and observe an inherent but heterogeneous fault tolerance. Building upon these insights, we develop an anomaly detection and clearance mechanism at the circuit level to eliminate outlier errors. At the model level, we propose a weight-rotation-enhanced planning algorithm to improve the fault tolerance of the LLM-based planner. Furthermore, we introduce an application-level technique, autonomy-adaptive voltage scaling, to dynamically adjust the operating voltage of the controllers. The voltage scaling circuit is co-designed to enable online voltage adjustment. Extensive experiments demonstrate that without compromising task quality, CREATE achieves 40.6% computational energy savings on average over nominal-voltage baselines and 35.0% over prior-art techniques. This further leads to 29.5% to 37.3% chip-level energy savings and approximately a 15% to 30% improvement in battery life.

Fully homomorphic encryption (FHE) allows operations to be performed directly on encrypted data, and has attracted massive attention in data security scenarios. Numerous resource efficient FHE acceleration methods have been proposed, including many on the optimization of modular multiplication (MM), a fundamental operation in FHE, by leveraging Karatsuba multiplication and NTT-friendly moduli for Montgomery MM (MMM). However, FHE is not yet practical due to its significant resource overheads. In this paper, we report an automated Karatsuba decomposition search strategy that drastically improves the efficiency of MM implementation. Our key idea is to integrate NTT-friendly moduli into Karatsuba decomposition within parallel MMM, and incorporate optimization features such as truncated multiplication ⌊AB/R⌋ and modular multiplicationABmodR. After a careful analysis on the optimization space, we propose an automated Karatsuba decomposition optimization search algorithm based on a greedy strategy to enhance efficiency and effectiveness. Theoretical analysis shows that, under the mainstream NTT-friendly modulus conditions, the optimized 2, 3, 4-term Karatsuba decomposition schemes achieve an average area reduction of 18% for MMM over the basic Karatsuba method and 57% over the classical Schoolbook method. Furthermore, hardware implementations on FPGA demonstrate 29% to 79% improvement, with an average of 59%, in Area/Throughput compared to the state-of-the-art implementations.

RRAM-based Deep Neural Network (DNN) accelerators have gained widespread usage in edge devices. However, the security vulnerabilities of RRAM-based accelerators hinder their real application. Current research on safeguarding RRAM-based accelerators predominantly relies on a single-level protection approach. This has resulted in the restriction of its protection scope, the rigidity and lack of generality in the protection method, or has had an impact on the computational efficiency of the system. As a result, it encounters substantial challenges in attaining comprehensive optimization across multiple dimensions, such as universality, the scope of protection, and security-related overheads. In this paper, we develop specific analyses on accelerators and attacks and build graph-based representations. Based on these, we partition the RRAM-based accelerators’ security into two levels: on-chip security and off-chip security. Furthermore, we propose a two-level protection framework for RRAM-based accelerators, which is calledOmniGuard. At the on-chip security level,OmniGuardproposes a bit-grained shuffle to achieve protection while using lightweight Benes Networks to maintain the CIM capability. At the off-chip level,OmniGuardproposes an RRAM-based AES engine to introduce the AES algorithm into the accelerator with significant acceleration and minimal overhead. Evaluation results demonstrate thatOmniGuardprovides powerful and flexible protection while achieving 1.33×∼4.38× speedup and 1.26×∼2.65× power savings, with only 5% energy overhead and 3% area overhead.

In the era of Big Data, enterprises and individuals often upload databases to the cloud for storage and querying, which involves the risk of data leakage. Encrypted databases based on fully homomorphic encryption (FHE) theoretically solve the leakage problem, but the actual deployment of such encrypted databases faces the challenge of high economic costs. Cloud service providers charge for data transfer volume and computation time. Unfortunately, FHE is very expensive in both aspects, with more than five orders of magnitude deterioration compared to directly transmitting and computing plaintext. In this paper, we present STEED, a low-cost encrypted database that tackles both bottlenecks simultaneously. In STEED, we first introduce a FHE framework called BatchPBS, a batch pro grammable bootstrapping framework that improves the recent Liu and Wang (ASIACRYPT 2023) amortised scheme from 6.7 ms to 3 msper ciphertext while adding multi-value bootstrapping (MVB) support. Based on BatchPBS, we propose efficient SQL algorithms in SIMD-style to reduce the computation time and a novel AES transcipher protocol to reduce the data transfer volume. Thus, STEED reduces query time by 13 × and data transfer amount by 165 to 534.9 × compared with SOTA work. Considering end-to-end economic cost of TPC-H query on a database with 1 million rows, STEED reduces the expense of deploying on AWS by $28444.8 per 100 queries. (The code can be found at https://github.com/alibaba-damo-academy/ctl-he)

Secure two-party neural network (2PC-NN) inference is a privacy-preserving inference method that protects the client's input and the server's model parameters. While addressing privacy concerns, it also incurs considerable over-heads. In this work, we propose Octopus, a faster and more communication-efficient 2PC-NN system than prior works. Octopus designs an optimized encoding method for fast homomorphic convolution, and further constructs homomorphic encryption-based convolutional computation protocol. Compared with the original coefficient encoding proposed by Cheetah, our method significantly reduces the resulting ciphertexts through packing output channels, thereby saving the communication cost and end-to-end execution time. Moreover, Octopus proposes an encoding-motivated fine tuning technique for convolutional neural networks, which fully utilizes the feature of coefficient encoding to adaptively adjust the neural network structure to maximize performance with negligible accuracy loss. We apply Octopus to the widely used model ResNet on CIFAR-10 and ImageNet dataset. Experiments illustrate that Octopus has obvious improvement compared with the state-of-the-art approaches, achieving a speedup of up to 2.75×, and reduces communication overhead by up to 7.19× for convolutions. As for secure inference, compared with Cheetah (resp., CrypTFlow2), Octopus demonstrates 1.41× (resp., 13.20×) lower communication cost and 1.25× (resp., 7.03×) faster execution time under a WAN setting.

Large language model (LLM)-based automatic speech recognition (ASR) has recently attracted a lot of attention due to its high recognition accuracy and enhanced multi-dialect support. However, the high decoding latency of LLMs challenges the real-time ASR requirements. Although speculative decoding has been explored for better decoding efficiency, they usually ignore the key characteristics of the ASR task and achieve limited speedup. To further reduce the real-time ASR latency, in this paper, we propose a novel speculative decoding framework specialized for ASR, dubbed SpecASR. SpecASR is developed based on our core observation that ASR decoding is audio-conditioned, which results in high output alignment between small and large ASR models, even given output mismatches in intermediate decoding steps. Therefore, SpecASR features an adaptive draft sequence generation process that dynamically modifies the draft sequence length to maximize the token acceptance length. SpecASR further proposes a draft sequence recycling strategy that reuses the previously generated draft sequence to reduce the draft ASR model latency. Moreover, a two-pass sparse token tree generation algorithm is also proposed to balance the latency of draft and target ASR models. With extensive experimental results, we demonstrate SpecASR achieves $3.04 \times-3.79 \times$ and $1.25 \times-1.84 \times$ speedup over the baseline autoregressive decoding and speculative decoding, respectively, without any loss in recognition accuracy.

The demand for efficient large language model (LLM) inference has propelled the development of dedicated accelerators. As accelerators are vulnerable to hardware faults due to aging, variation, etc, existing accelerator designs often reserve a large voltage margin or leverage algorithm-based fault tolerance (ABFT) techniques to ensure LLM inference correctness. However, previous methods often overlook the inherent fault tolerance of LLMs, leading to high computation and energy overhead. To enable reliable yet efficient LLM inference, in this paper, we propose a novel algorithm/circuit co-design framework, dubbed ReaLM. For the first time, we systematically characterize the fault tolerance of LLMs by performing a large-scale error injection study of representative LLMs and natural language understanding tasks. Then, we propose a statistical ABFT algorithm that fully leverages the error robustness to minimize error recovery as much as possible. We also customize the error detection circuits to enable a low-cost online collection of error statistics. Extensive experiments show that with only 1.42% circuit area and 1.79% power overhead, our ReaLM can reduce perplexity degradation from 18.54 to 0.29. Compared to existing methods, ReaLM consistently reduces recovery costs across different operating voltages and improves energy efficiency by up to 35.83% without compromising LLM performance. Our error injection code is available at https://github.com/PKU-SEC-Lab/ReaLM_DAC25/.

Transformer-based large language models (LLMs) have achieved impressive performance in various natural language processing (NLP) applications. However, the high memory and computation cost induced by the KV cache limits the inference efficiency, especially for long input sequences. Compute-in-memory (CIM)-based accelerators have been proposed for LLM acceleration with KV cache pruning. However, as existing accelerators only support static pruning with a fixed pattern or dynamic pruning with primitive implementations, they suffer from either high accuracy degradation or low efficiency. In this paper, we propose a ferroelectric FET (FeFET)-based unified content addressable memory (CAM) and CIM architecture, dubbed as UniCAIM. UniCAIM features simultaneous support for static and dynamic pruning with 3 computation modes: 1) in the CAM mode, UniCAIM enables approximate similarity measurement in $\mathcal{O}(1)$ time for dynamic KV cache pruning with high energy efficiency; 2) in the charge-domain CIM mode, static pruning can be supported based on accumulative similarity score, which is much more flexible compared to fixed patterns; 3) in the current-domain mode, exact attention computation can be conducted with a subset of selected KV cache. We further propose a novel CAM/CIM cell design that leverages the multi-level characteristics of FeFETs for signed multibit storage of the KV cache and in-place attention computation. With extensive experimental results, we demonstrate UniCAIM can reduce the area-energy-delay product (AEDP) by $8.2 \sim 831 \times$ over the state-of-the-art CIM-based LLM accelerators at the circuit level, along with high accuracy comparable with dense attention at the application level, showing its great potential for efficient long-context LLM inference.

The Mixture of Experts (MoE) architecture has demonstrated significant advantages as it enables to increase the model capacity without a proportional increase in computation. However, the large MoE model size still introduces substantial memory demands, which usually requires expert offloading on resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU inference has been proposed to leverage CPU computation to reduce expert loading overhead but faces major challenges: on one hand, the expert activation patterns of MoE models are highly unstable, rendering the fixed mapping strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU schedule for MoE is inherently complex due to the diverse expert sizes, structures, uneven workload distribution, etc. To address these challenges, in this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that improves resource utilization through a novel CPU-GPU scheduling and cache management system. HybriMoE introduces (i) a dynamic intra-layer scheduling strategy to balance workloads across CPU and GPU, (ii) an impact-driven inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to mitigate expert activation instability. We implement HybriMoE on top of the kTransformers framework and evaluate it on three widely used MoE-based LLMs. Experimental results demonstrate that HybriMoE achieves an average speedup of $\mathbf{1. 3 3} \times$ in the prefill stage and $1.70 \times$ in the decode stage compared to state-of-the-art hybrid MoE inference framework. Our code is available at: https://github.com/PKU-SEC-Lab/HybriMoE.

Deep neural networks for image super-resolution (SR) have demonstrated superior performance. However, the large memory and computation consumption hinders their deployment on resource-constrained devices. Binary neural networks (BNNs), which quantize the floating point weights and activations to 1-bit can significantly reduce the cost. Although BNNs for image classification have made great progress these days, existing BNNs for SR still suffer from a large performance gap between the FP SR networks. To this end, we observe the activation distribution in SR networks and find much larger pixel-to-pixel, channel-to-channel, layer-to-layer, and image-to-image variation in the acti-vation distribution than image classification networks. However, existing BNNs for SR fail to capture these variations that contain rich information for image reconstruction, leading to inferior performance. To address this problem, we propose SCALES, a binarization method for SR networks that consists of the layer-wise scaling factor, the spatial re-scaling method, and the channel-wise re-scaling method, capturing the layer-wise, pixel-wise, and channel-wise variations efficiently in an input-dependent manner. We evaluate our method across different network architectures and datasets. For CNN-based SR networks, our binarization method SCALES outperforms the prior art method by 0.2dB with fewer parameters and operations. With SCALES, we achieve the first accurate binary Transformer-based SR network, improving PSNR by more than 1dB compared to the baseline method.

Lookup tables (LUTs) are widely used for reconfigurable computing applications due to the capability of implementing arbitrary logic functions. Various emerging non-volatile memories (eNVMs) have been introduced for LUT designs with reduced hardware cost and power consumption compared with conventional SRAM-based LUT. However, the existing designs still follow the conventional LUT architecture, where the memory cells are only used for storage of configuration bits, requiring dedicated bulky multiplexer (MUX) for computation of each LUT, resulting in inevitable high area, latency, and energy cost. In this work, a compact and efficient non-volatile LUT architecture based on ferroelectric FET (FeFET) array is proposed, where the configuration bit storage and computation can be implemented within the FeFET array through in-situ combinatorial one-hot encoding, eliminating the need of costly MUX for each LUT. Moreover, multibit LUTs can be efficiently implemented in the FeFET array using only one shared decoder instead of multiple costly MUXs. Due to the eliminated MUX in the calculation path, the proposed LUT can also achieve enhanced computation speed compared with the conventional LUTs. Based on the proposed LUT architecture, the input expansion of LUT, full adder, and content addressable memory are further implemented and demonstrated with reduced hardware and energy cost. Evaluation results show that the proposed FeFET array-based LUT architecture achieves 51.7×/8.3× reduction in area-energy-delay product compared with conventional SRAM-based/FeFET-based LUT architecture, indicating its great potential for reconfigurable computing applications.

Private convolutional neural network (CNN) inference based on hybrid homomorphic encryption (HE) and two-party computation (2$P$C) emerges as a promising technique for sensitive user data protection. However, homomorphic convolutions (HConvs) suffer from high computation costs due to the extensive number theoretic transforms (NTTs). While customized accelerators have been proposed, they usually overlook the intrinsic error resilience and native sparsity of DNNs and hybrid HE/2$P$C protocols. In this paper, we propose FLASH, leveraging these key characteristics for highly efficient HConv. Specifically, we observe the private DNN inference is robust to computation errors and propose approximate fast Fourier transforms (FFTs) to replace NTTs and avoid the expensive modular reduction operations. We also design a flexible sparse FFT dataflow leveraging the high sparsity of weight plaintexts. With extensive experiments, we demonstrate FLASH improves the power efficiency by 90.7× for weight transforms and by 9.7× for all transforms in HConvs compared to existing works. As for the HConvs in ResNet-18 and ResNet-50, FLASH achieves about 87.3% energy consumption reduction.

Heavily constrained edge-side tasks necessitate AI chips with low power consumption, low latency, and low cost. In recent years, digital compute-in-memory (CIM) has emerged as a promising solution to enhance energy efficiency and throughput density. However, digital CIM still faces various challenges: significant power and area overhead from multiplication, difficulty in exploiting fine-grained sparsity, and throughput degradation associated with bit-serial architecture. In this work, we propose Adder-DCIM: an efficient parallel bit-flexible DCIM accelerator joint model compression framework for AdderNet inference, in which the key contributions are: 1) a CIM-friendly model compression framework that includes operator decomposition, lossless fine-grained sparsity, and Kullback-Leibler-divergence(KLD)-based Cin-wise mixed-precision quantization; 2) a synchronous parallel DCIM architecture for throughput improvement with mix-precision quantization; 3) a bit-flexible minimal selector circuit for efficient mixed-precision computation. The experimental results demonstrate that under a 28-nm process, the proposed Adder-DCIM achieves a peak energy efficiency of 134 TOPS/W and a peak throughput density of 6.49 TOPS/mm2at INT8. When running ResNet20 on CIFAR10 and ResNet50 on ImageNet, the proposed Adder-DCIM achieves 255 TOPS/[email protected] and 243 TOPS/[email protected] with only a slight decrease in accuracy by 0.34% and 0.7%, respectively. Compared to multiply-based DCIM, Adder-DCIM improves energy efficiency × throughput density metrics by 20.7× for ResNet50 inference.

Large language models (LLMs) have demonstrated remarkable proficiency in a wide range of natural language processing applications. However, the high energy and latency overhead induced by the KV cache limits the edge deployment, especially for long contexts. Emerging hybrid bonding (HB) technology has been proposed as a promising alternative to conventional near-memory processing (NMP) architectures, offering improved bandwidth efficiency and lower power consumption while exhibiting characteristics of distributed memory.In this paper, we propose H2EAL, an HB-based accelerator with sparse attention algorithm-hardware co-design for efficient LLM inference at the edge. At the algorithm level, we propose a hybrid sparse attention scheme with static and dynamic sparsity for different heads to fully leverage the sparsity with high accuracy. At the hardware level, we co-design the hardware to support hybrid sparse attention and propose memory-compute co-placement to address the distributed memory bottleneck. Since different attention heads exhibit different sparse patterns and the attention structure often mismatches the HB architecture, we further develop a load-balancing scheduler with parallel tiled attention to address workload imbalance and optimize the mapping strategy. Extensive experiments demonstrate H2EAL achieves 5.20 ∼ 48.21× speedup and 6.22 ∼ 73.48× energy efficiency improvement over baseline HB implementation, with a negligible average accuracy drop of 0.87% on multiple benchmarks.

Large Language Models (LLMs) with Mixture-of-Expert (MoE) architectures achieve superior model performance with reduced computation costs, but at the cost of high memory capacity and bandwidth requirements. Near-Memory Processing (NMP) accelerators that stack memory directly on the compute through hybrid bonding have demonstrated high bandwidth with high energy efficiency, becoming a promising architecture for MoE models. However, as NMP accelerators comprise distributed memory and computation, how to map the MoE computation directly determines the LLM inference efficiency. Existing parallel mapping strategies, including Tensor Parallelism (TP) and Expert Parallelism (EP), suffer from either high communication costs or unbalanced computation utilization, leading to inferior efficiency. The dynamic routing mechanism of MoE LLMs further aggravates the efficiency challenges. Therefore, in this paper, we propose HD-MoE to automatically optimize the MoE parallel computation across an NMP accelerator. HD-MoE features an offline automatic hybrid parallel mapping algorithm and an online dynamic scheduling strategy to reduce the communication costs while maximizing the computation utilization. With extensive experimental results, we demonstrate that HD-MoE achieves a speedup ranging from 1.1× to 1.8× over TP, 1.1× to 1.5× over EP, and 1.0× to 1.4× over the baseline Hybrid TP-EP with Compute-Balanced parallelism strategies.

Private transformer inference combining hybrid homomorphic encryption (HE) and multi-party computation (MPC) protocols has attracted increasing attention. By leveraging oblivious transfer (OT) and HE for nonlinear and linear operations, respectively, it enables accurate computation with formal privacy protection. However, existing works suffer from significant computation cost due to suboptimal protocol integration and hardware inefficiencies. We propose FENIX, a protocol-hardware co-design framework that significantly accelerates hybrid HE/MPC execution. At the protocol level, FENIX proposes the fine-grained OT partitioning to overlap HE and OT operations and minimize computational stalls. Besides, we introduce a flexible batch encoding for computation-storage trade-offs. At the hardware level, we identify memory-bound computations of both HE and OT as the critical bottleneck and offload them to near-memory processing (NMP) to reduce latency. Above all, FENIX demonstrates a flexible framework with significant latency reduction and performance improvement in private inference.

The growing demand for efficient long-sequence modeling on edge devices has propelled widespread adoption of State Space Models (SSMs) like Mamba, due to their superior computational efficiency and scalability. As its autoregressive generation process remains memory-bound, speculative decoding has been proposed that incorporates draft model generation and target model verification. However, directly applying speculative decoding to SSMs faces three key challenges: (1) hidden state backtracking difficulties, (2) tree-based parallel verification incompatibility, and (3) hardware workload mismatch. To address these challenges, we propose SpecMamba, the first FPGA-based accelerator for Mamba with speculative decoding, which features system, algorithm, and hardware co-design. At the system level, we present a memory-aware hybrid backtracking strategy to coordinate both models. At the algorithm level, we propose first-in-first-out (FIFO)-based tree verification with tiling to minimize memory access. At the hardware level, we customize a dataflow that computes linear layers in parallel and SSM layers in series to enable maximal overlapping. Implemented on AMD FPGA platforms (VHK158 and VCK190), SpecMamba achieves a 2.27× speedup over GPU baselines and a 2.85× improvement compared to prior FPGA solutions, while demonstrating 5.41× and 1.26× higher energy efficiency, respectively.

Private large language model (LLM) inference based on secure multi-party computation (MPC) achieves formal data privacy protection but suffers from significant latency overhead, especially for long input sequences. While key-value (KV) cache eviction and sparse attention algorithms have been proposed for efficient LLM inference in plaintext, they are not designed for MPC and cannot benefit private LLM inference directly. In this paper, we propose an accurate and MPC-friendly KV cache eviction framework, dubbed MPCache, building on the observation that historical tokens in a long sequence may have different effects on the downstream decoding. Hence, MPCache combines a look-once static eviction algorithm to discard unimportant KV cache and a query-aware dynamic selection algorithm to activate only a small subset of KV cache for attention computation. MPCache further incorporates a series of optimizations for efficient dynamic KV cache selection, including MPC-friendly similarity approximation, hierarchical KV cache clustering, and cross-layer index-sharing strategy. Extensive experiments demonstrate that MPCache consistently outperforms prior-art KV cache eviction baselines across different generation tasks and achieves 1.8 ~ 2.01x and 3.39 ~ 8.37x decoding latency and communication reduction on different sequence lengths, respectively.

Fully homomorphic encryption over torus (TFHE) enables the execution of arbitrary functions on encrypted data through programmable bootstrapping (PBS). However, performing all operations on ciphertext during PBS results in high computational and memory requirements, limiting the deployment of PBS in real-world scenarios. Previous TFHE accelerator designs have attempted to improve performance by employing specific dataflow and functional units, but these techniques may require large off-chip bandwidth or on-chip storage when scaling up computation capacity. Additionally, the design of specialized functional units may limit the utilization of computation units when facing dynamic secure parameter settings. To address these challenges and further improve PBS throughput in TFHE, we propose Matrix , an ASIC-based architecture that balances off-chip bandwidth and on-chip storage according to the execution flow of PBS. In Matrix , we utilize a unified special-prime-based processing element (PE) that achieves high utilization with minimal resource overhead. Furthermore, we propose a hybrid PBS dataflow that can efficiently reduce computation complexity and memory requirements. Compared to state-of-the-art TFHE accelerators, Matrix achieves 1.43 × -5.66 × throughput improvement for PBS. For ZAMA Deep-NN benchmark, we achieve 525.60× and 68.06× speedup compared to CPU and GPU, respectively. 1

With the widespread use of machine learning (ML), privacy concerns during neural network inference are attracting growing attention. Secure two-party neural network (2PC-NN) inference is the privacy-preserving inference method, which allows client to obtain the inference result without disclosing client’s input to the server. The server’s model parameters are also confidential to the client. However, current 2PC-NN inference schemes still have large overhead, especially for non-linear functions. In this paper, we present Swift, a fast secure 2PC-NN inference scheme based on fully homomorphic encryption (FHE) and secret sharing (SS). FHE protects the input and model parameters in linear functions, while SS is integrated to protect the non-linear functions. Concretely, Swift integrates FHE and SS to design secure and efficient non-linear protocols used for ReLU and max pooling. To further optimize performance, Swift employs FHE with computation-friendly coefficient encoding for fast execution of linear functions, and SIMD encoding for non-linear functions. Swift constructs efficient encoding conversion protocol between the coefficient-encoded ciphertext and the SIMD-encoded ciphertext. Finally, Swift achieves secure neural network inference framework for MNIST dataset. Compared with Cheetah (USENIX 2022), the execution time of ReLU, max pooling, secure inference under a WAN setting improves$7.4\times $,$13.3\times $,$1.9\times $, respectively.

Fusing 3D LiDAR features with 2D camera features is a promising technique for enhancing the accuracy of 3D detection, thanks to their complementary physical properties. While most of the existing methods focus on directly fusing camera features with raw LiDAR point clouds or shallow-level 3D features, it is observed that directly combining 2D and 3D features in deeper layers actually leads to a decrease in accuracy due to feature misalignment. The misalignment, which stems from the aggregation of features learned from large receptive fields, becomes increasingly more severe as we delve into deeper layers. In this paper, we propose PathFusion as a solution to enable the alignment of semantically coherent LiDAR-camera deep feature fusion. PathFusion introduces a path consistency loss at multiple stages within the network, encouraging the 2D backbone and its fusion path to transform 2D features in a way that aligns semantically with the transformation of the 3D backbone. This ensures semantic consistency between 2D and 3D features, even in deeper layers, and amplifies the usage of the network’s learning capacity. We apply PathFusion to improve a prior-art fusion baseline, Focals Conv, and observe an improvement of over 1.2% in mAP on the nuScenes test split consistently with and without testing-time data augmentations, and moreover, PathFusion also improves KITTI AP3D(R11) by about 0.6% on the moderate level.

There has been a growing trend in deploying deep neural networks (DNNs) on tiny devices. However, deploying DNNs on such devices poses significant challenges due to the contradiction between DNNs' substantial memory requirements and the stringent memory constraints of tiny devices. Some prior works incur large latency overhead to save memory and target only simple CNNs, while others employ coarse-grained scheduling for complicated networks, leading to limited memory footprint reduction. This paper proposes MoteNN that performs fine-grained scheduling via operator partitioning on arbitrary DNNs to dramatically reduce peak memory usage with little latency overhead. MoteNN presents a graph representation named Axis Connecting Graph (ACG) to perform operator partition at graph-level efficiently. MoteNN further proposes an algorithm that finds the partition and schedule guided by memory bottlenecks. We evaluate MoteNN using various popular networks and show that MoteNN achieves up to 80% of peak memory usage reduction compared to the state-of-art works with nearly no latency overhead on tiny devices.

With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic encryption (HE) has been proposed to protect user query privacy. However, private embedding table query has to be formulated as a HE-based matrix-vector multiplication problem and suffers from enormous computation and communication overhead. We observe the overhead mainly comes from the neglect of 1) the one-hot nature of user queries and 2) the robustness of the embedding table to low bit-width quantization noise. Hence, in this paper, we propose a private embedding table query optimization framework, dubbed FastQuery. FastQuery features a communication-aware embedding table quantization algorithm and a one-hot-aware dense packing algorithm to simultaneously reduce both the computation and communication costs. Compared to prior-art HE-based frameworks, e.g., Cheetah, Iron, and Bumblebee, FastQuery achieves more than 4.3×, 2.7×, 1.3× latency reduction, respectively and more than 75.7×, 60.2×, 20.2× communication reduction, respectively, on both LLAMA-7B and LLAMA-30B.

The use of cross-scheme fully homomorphic encryption (FHE) in privacy-preserving applications present to be a new challenge to hardware accelerator design. Existing accelerator architectures with customized polynomial-level operator abstraction fail to efficiently handle hybrid FHE schemes due to the mismatch between computational demands and available hardware resources under various parameter settings. In this work, we propose a new accelerator architecture that consists of a novel finer-grained low-level operator, i.e., Meta-OP, that not only mathematically supports a diverse range of polynomial operations, but is also hardware-friendly for accelerator design without complex topological logic. We then design a new slot-based data management scheme to efficiently handle the distinct memory access patterns over the Meta-OP. With a slot-based data management approach, Alchemist can accelerate both arithmetic and logic FHE workloads with high hardware utilization rates. In the experiment, we show that Alchemist is up to 24,829X faster than CPU. For arithmetic FHE, compared with the SOTA ASIC accelerators, Alchemist achieves a 29.4X performance per area improvement on average. For logic FHE, compared with the SOTA ASIC accelerators, Alchemist achieves a 7.0X overall speed up on average.

Stochastic computing (SC) has emerged as a promising computing paradigm for neural acceleration. However, how to accelerate the state-of-the-art Vision Transformer (ViT) with SC remains unclear. Unlike convolutional neural networks, ViTs introduce notable compatibility and efficiency challenges because of their nonlinear functions, e.g., softmax and Gaussian Error Linear Units (GELU). In this paper, for the first time, a ViT accelerator based on end-to-end SC, dubbed ASCEND, is proposed. ASCEND co-designs the SC circuits and ViT networks to enable accurate yet efficient acceleration. To overcome the compatibility challenges, ASCEND proposes a novel deterministic SC block for GELU and leverages an SC-friendly iterative approximate algorithm to design an accurate and efficient softmax circuit. To improve inference efficiency, ASCEND develops a two-stage training pipeline to produce accurate low-precision ViTs. With extensive experiments, we show the proposed GELU and softmax blocks achieve 56.3% and 22.6% error reduction compared to existing SC designs, respectively, and reduce the area-delay product (ADP) by 5.29 x and 12.6 x, respectively. Moreover, compared to the baseline low-precision ViTs, ASCEND also achieves significant accuracy improvements on CIFARI0 and CIFARI00.

Vision Transformer (ViT) acceleration with field programmable gate array (FPGA) is promising but challenging. Existing FPGA-based ViT accelerators mainly rely on temporal architectures, which process different operators by reusing the same hardware blocks and suffer from extensive memory access overhead. Pipelined architectures, either coarse-grained or fine-grained, unroll the ViT computation spatially for memory access efficiency. However, they usually suffer from significant hardware resource constraints and pipeline bubbles induced by the global computation dependency of ViT. In this paper, we introduce HG-PIPE, a pipelined FPGA accelerator for high-throughput and low-latency ViT processing. HG-PIPE features a hybrid-grained pipeline architecture to reduce on-chip buffer cost and couples the computation dataflow and parallelism design to eliminate the pipeline bubbles. HG-PIPE further introduces careful approximations to implement both linear and non-linear operators with abundant Lookup Tables (LUTs), thus alleviating resource constraints. With a VCK190 FPGA, HG-PIPE realizes end-to-end ViT acceleration on a single device and achieves 7118 images/s, which is 2.81× faster than a V100 GPU.

End-to-end stochastic computing (SC) emerges as a promising paradigm for efficient neural acceleration. However, existing serial SC accelerators face serious accuracy challenges due to errors in addition, limited activation compatibility, and limited bitstream multiplication. In this paper, we propose OSCA, an accurate yet efficient end-to-end serial SC accelerator. OSCA features fine-grained scaling control in the SC adder to minimize quantization error and introduces novel segment units for piecewise approximation of complex nonlinear activation functions. We also design an integer-bitstream multiplier that guarantees accuracy regardless of the correlation between input bitstreams. Compared to the serial baseline accelerator of INT8 precision, we can achieve a 30.18% reduction in root mean square error (RMSE) along with an 87.75% area-delay product (ADP) reduction. Moreover, compared to a more accurate parallel accelerator, OSCA achieves a 92.05% ADP reduction with a 12.54× improvement in area efficiency, while also increasing inference accuracy by 0.44% and 0.86%.

Private deep neural network (DNN) inference based on secure two-party computation (2PC) enables secure privacy protection for both the server and the client. However, existing secure 2PC frameworks suffer from a high inference latency due to enormous communication. As the communication of both linear and non-linear DNN layers reduces with the bit widths of weight and activation, in this paper, we propose PrivQuant, a framework that jointly optimizes the 2PC-based quantized inference protocols and the network quantization algorithm, enabling communication-efficient private inference. PrivQuant proposes DNN architecture-aware optimizations for the 2PC protocols for communication-intensive quantized operators and conducts graph-level operator fusion for communication reduction. Moreover, PrivQuant also develops a communication-aware mixed precision quantization algorithm to improve the inference efficiency while maintaining high accuracy. The network/protocol co-optimization enables PrivQuant to outperform prior-art 2PC frameworks. With extensive experiments, we demonstrate PrivQuant reduces communication by 11×, 2.5 × and 2.8×, which results in 8.7×, 1.8 × and 2.4× latency reduction compared with SiRNN, COINN, and CoPriv, respectively.

In this paper, we propose MCUBERT to enable language models like BERT on tiny microcontroller units (MCUs) through network and scheduling co-optimization. We observe the embedding table contributes to the major storage bottleneck for tiny BERT models. Hence, at the network level, we propose an MCU-aware two-stage neural architecture search algorithm based on clustered low-rank approximation for embedding compression. To reduce the inference memory requirements, we further propose a novel fine-grained MCU-friendly scheduling strategy. Through careful computation tiling and re-ordering as well as kernel design, we drastically increase the input sequence lengths supported on MCUs without any latency or accuracy penalty. MCUBERT reduces the parameter size of BERT-tiny and BERT-mini by 5.7× and 3.0× and the execution memory by 3.5× and 4.3×, respectively. MCUBERT also achieves 1.5× latency reduction. For the first time, MCUBERT enables lightweight BERT models on commodity MCUs and processing more than 512 tokens with less than 256KB of memory.

Secure two-party computation (2PC) based on homomorphic encryption (HE) achieves formal data privacy protection and gets increasing adoption for private deep neural network (DNN) inference. As modern HE schemes usually operate on polynomials, existing works rely on manually-designed HE kernels for representative DNN operations. However, this is not only unscalable considering the diverse operator types, shapes, polynomial orders, etc, but also misses important optimization opportunities. In this paper, we introduce FlexHE, a flexible kernel generation framework to enable automatic generation and optimization of HE kernels for 2PC-based private inference. Given a high-level description of DNN operations, FlexHE can systematically define the HE kernel design space considering various optimization dimensions, including loop tiling, reordering, etc. We also analyze the communication and computation impact of different optimization dimensions for design space reduction. To search for the best kernel design, a two-level optimization problem is formulated and iteratively solved with an integer linear programming (ILP) formulation. With extensive experimental results, we not only demonstrate a better coverage of DNN operations including depth-wise Conv3D and dilated Conv3D, but also achieve more than 100×, 7.9×, and 4.2× latency reduction compared to prior-art HElayers, Cheetah, and Falcon, respectively.

Mixture-of-Experts (MoE) models are designed to enhance the efficiency of large language models (LLMs) without proportionally increasing the computational demands. However, their deployment on edge devices still faces significant challenges due to high on-demand loading overheads from managing sparsely activated experts. This paper introduces AdapMoE, an algorithm-system co-design framework for efficient MoE inference. AdapMoE features adaptive expert gating and management to reduce the on-demand loading overheads. We observe the heterogeneity of experts loading across layers and tokens, based on which we propose a sensitivity-based strategy to adjust the number of activated experts dynamically. Meanwhile, we also integrate advanced prefetching and cache management techniques to further reduce the loading latency. Through comprehensive evaluations on various platforms, we demonstrate AdapMoE consistently outperforms existing techniques, reducing the average number of activated experts by 25% and achieving a 1.35× speedup without accuracy degradation. Code is available at: https://github.com/PKU-SEC-Lab/AdapMoE.

Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is constrained by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verification, e.g., Medusa, has been proposed to improve decoding parallelism and efficiency, it often struggles with maintaining contextual relationships due to its independent token prediction approach and incurs significant verification overhead, especially with large tree sizes and batch processing. In this paper, we propose ProPD, an efficient LLM parallel decoding framework based on dynamic token tree pruning and generation. ProPD features an advanced early pruning mechanism to efficiently eliminate unpromising token sequences to improve verification efficiency. Additionally, it introduces a dynamic token tree generation algorithm to balance the computation and parallelism of the verification phase in real-time and maximize the overall efficiency across different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD consistently outperforms existing decoding algorithms by 1.1--3.2 ×.

Fully Homomorphic Encryption (FHE) is crucial for privacy-preserving computing, which allows direct computation on encrypted data. While various FHE schemes have been proposed, none of them efficiently support both arithmetic FHE and logic FHE simultaneously. To address this issue, researchers explore the combination of different FHE schemes within a single application and propose algorithms for the conversion between them. Unfortunately, all prior ASIC-based FHE accelerators are designed to support a single FHE scheme, and none of them supports the acceleration for FHE scheme conversion. This necessitates FHE acceleration systems to integrate multiple accelerators for different schemes, leading to increased system complexity and hindering performance enhancement. In this paper, we present the first multi-modal FHE accelerator based on a unified architecture, which efficiently supports CKKS, TFHE, and their conversion scheme within a single accelerator. To achieve this goal, we first analyze the theoretical foundations of the aforementioned schemes and highlight their composition from a finite number of arithmetic kernels. Then, we investigate the challenges for efficiently supporting these kernels within a unified architecture, which include 1) concurrent support for NTT and FFT, 2) maintaining high hardware utilization across various polynomial lengths, and 3) ensuring consistent performance across diverse arithmetic kernels. To tackle these challenges, we propose a novel FHE accelerator named Trinity, which in-corporates algorithm optimizations, hardware component reuse, and dynamic workload scheduling to enhance the acceleration of CKKS, TFHE, and their conversion scheme. By adaptive select the proper allocation of components for NTT and MAC, Trinity maintains high utilization across NTTs with various polynomial lengths and imbalanced arithmetic workloads. The experiment results show that, for the pure CKKS and TFHE workloads, the performance of our Trinity outperforms the state-of-the- art accelerator for CKKS (SHARP) and TFHE (Morphling) by 1.49 x and 4.23 x, respectively. Moreover, Trinity achieves 919.3 x performance improvement for the FHE-conversion scheme over the CPU-based implementation. Notably, despite the performance improvement, the hardware overhead of Trinity is only 85 % of the summed circuit areas of SHARP and Morphling.

Large Language Models (LLMs) are widely used in today's tasks of natural language processing. To support applications like multi-turn chats, document understanding, and content generation, models with long context lengths are growing in importance. However, managing long contexts brings substantial challenges due to the expansion of key-value cache (KV cache). Longer KV cache requires larger memory, limiting the batch-size thus decreasing throughput. Also, computing attention over long KV cache incurs more memory access, hurting the end-to-end latency. Prior works find that it is sufficient to use only the recent and high-impact tokens for attention computation, allowing the eviction of less vital tokens to shrink cache size. Nonetheless, we observe a dynamic shift in token importance across different decoding steps. Tokens initially evicted might regain importance after certain decoding steps. To address this, we propose ArkVale, a page-based KV cache manager that can recognize and recall currently important tokens evicted before. We asynchronously copy the filled page into external memory (e.g., CPU memory) as backup and summarize it into a much smaller digest by constructing the bounding-volume of its keys. Before attention computation, we measure all pages' importance based on their digests, recall the important ones, evict the unimportant ones, and select the top-ranked pages for attention computation. Experiment results show that ArkVale performs well on various long context tasks with negligible accuracy loss under 2k$\sim$4k cache budget and can improve decoding latency to $2.2\times$ and batching throughput to $4.6\times$ because it applies attention on only a small subset of pages and reduce per-sample memory usage of KV cache.

Homomorphic encryption (HE)-based deep neural network (DNN) inference protects data and model privacy but suffers from significant computation overhead. We observe transforming the DNN weights into circulant matrices converts general matrix-vector multiplications into HE-friendly 1-dimensional convolutions, drastically reducing the HE computation cost. Hence, in this paper, we propose PrivCirNet, a protocol/network co-optimization framework based on block circulant transformation. At the protocol level, PrivCirNet customizes the HE encoding algorithm that is fully compatible with the block circulant transformation and reduces the computation latency in proportion to the block size. At the network level, we propose a latency-aware formulation to search for the layer-wise block size assignment based on second-order information. PrivCirNet also leverages layer fusion to further reduce the inference cost. We compare PrivCirNet with the state-of-the-art HE-based framework Bolt (IEEE S\&P 2024) and HE-friendly pruning method SpENCNN (ICML 2023). For ResNet-18 and Vision Transformer (ViT) on Tiny ImageNet, PrivCirNet reduces latency by $5.0\times$ and $1.3\times$ with iso-accuracy over Bolt, respectively, and improves accuracy by $4.1$\% and $12$\% over SpENCNN, respectively. For MobileNetV2 on ImageNet, PrivCirNet achieves $1.7\times$ lower latency and $4.2$\% better accuracy over Bolt and SpENCNN, respectively. Our code and checkpoints are available on Git Hub.

Layer Pipeline (LP) represents an innovative architecture for neural network accelerators, which implements task-level pipelining at the granularity of layers. Despite improvements in throughput, LP architectures face challenges due to complicated dataflow design, intricate design space and high resource requirements. In this paper, we introduce an accelerator synthesis framework, CASCADE. CASCADE leverages a novel dataflow, CARD, to efficiently manage convolutional operations’ irregular memory access patterns using simplified logic and minimal buffers. It also employs advanced design space exploration methods to optimize unrolling parallelism and FIFO depth settings automatically for each layer. Finally, to further enhance resource efficiency, CASCADE leverages Lookup Table-based multiplication and accumulation units. With extensive experimental results, we demonstrate that CASCADE significantly outperforms existing works, achieving a$3\times $improvement in resource efficiency and a$4\times $improvement in power efficiency. It achieves over$1.5\times 10^{4}$frames per second throughput and 71.9% accuracy on ImageNet.

Edge artificial intelligence applications impose rigorous demands on local hardware to improve throughput and energy efficiency. Computing-in-memory (CIM) architectures provide high parallel and energy-efficient solutions to accelerate the multiply-and-accumulate (MAC) operations in neural networks (NNs). While SRAM-based charge-domain CIM is achieving thousands of TOPS/W energy efficiency, it encounters limitations when dealing with full NN model deployments where both activations and weights are signed. This paper proposes an SRAM-based signed batch normalization (BN) CIM macro for supporting efficient bitwise sparse MAC computation with signed operands and BN operations in deep neural networks. The key features of this macro encompass: 1) a multibit weight unit for the optimization of bitstream sparsity and the sign bit computation, 2) a 2b-serial input configuration to increase throughput and the ADC energy amortization, and 3) a quantization-hardware co-design for the BN implementation. Measurement results show that the proposed 28 nm 64 Kb CIM macro achieves 16.38 TOPS throughput and 4.55 POPS/W energy efficiency, both normalized to 1b operands. The test accuracy of CIFAR10 is 92%, based on the ResNet18 model with co-design BN implementation at signed-8b precision activations and weights.

End-to-end stochastic computing (SC) enables fault-tolerant and area-efficient neural acceleration by conducting non-linear addition, including accumulation and activation functions, in SC bitstreams. However, existing non-linear adder designs suffer from a high hardware cost, accounting for a major portion of the datapath power and area, and may also have limited computation accuracy and flexibility. In this paper, we propose an accurate yet efficient non-linear adder design. We analyze the redundancy in existing designs and propose a parameterized approximate non-linear adder design space. By systematic design space exploration, we develop non-linear adders that are significantly more efficient than existing designs with negligible computation error. We further propose a spatial-temporal architecture to improve the design flexibility and efficiency for a wide range of network sizes. To support state-of-the-art networks, e.g., ResNet18, we demonstrate that our design can reduce the datapath area by 2.16× compared with the baseline designs. Our design can also reduce the area-delay product (ADP) of the non-linear adder by 4.13× and 23.29× for large and small convolution layers in ResNet18, respectively.

Stochastic computing (SC) emerges as a fault-tolerant and area-efficient computing paradigm for neural acceleration. However, existing SC accelerators suffer from an intrinsic trade-off between inference accuracy and efficiency: accurate SC re-quires high precision computation but suffers from an exponential increase of bitstream length and inference latency. In this paper, we discover the high precision residual as a key remedy and propose to combine a low precision datapath with a high precision residual to improve inference accuracy with minimum efficiency overhead. We also propose to fuse batch normalization with the activation function to further improve the inference efficiency. The effectiveness of our proposed method is verified on a recently proposed SC accelerator. With extensive results, we show that our proposed SC-friendly network achieves 9.43% accuracy im-provements compared to the baseline low precision networks with only 1.3% area-delay product (ADP) increase. We further show$\boldsymbol{3.01\times}$ADP reduction compared to the baseline SC accelerator with almost iso-accuracy.

Deep neural networks (DNNs) have revolutionized different applications ranging from computer vision to natural language processing, and are widely deployed in data centers and edge devices. It can be foreseen that DNNs will be applied in more and more safety-critical applications like autonomous driving and robotics, which typically require highly reliable computing to avoid catastrophic consequences. Therefore, not only the model's robustness against various perturbations like adversarial noise, but also the robustness of the silicon-based accelerators to hardware faults needs to be comprehensively investigated [1], [2].

Efficient networks, e.g., MobileNetV2, EfficientNet, etc, achieves state-of-the-art (SOTA) accuracy with lightweight computation. However, existing homomorphic encryption (HE)-based two-party computation (2PC) frameworks are not op-timized for these networks and suffer from a high inference overhead. We observe the inefficiency mainly comes from the packing algorithm, which ignores the computation character-istics and the communication bottleneck of homomorphically encrypted depthwise convolutions. Therefore, in this paper, we propose Falcon, an effective dense packing algorithm for HE-based 2PC frameworks. Falcon features a zero-aware greedy packing algorithm and a communication-aware operator tiling strategy to improve the packing density for depth wise convo-lutions. Compared to SOTA HE-based 2PC frameworks, e.g., CrypTFlow2, Iron and Cheetah, Falcon achieves more than 15.6 x, 5.1 x and 1.8 x latency reduction, respectively, at operator level. Meanwhile, at network level, Falcon allows for 1.4 % and 4.2% accuracy improvement over Cheetah on CIFAR-100 and Tiny Imagenet datasets with iso-communication, respecitvely.

With the rapid advancements of deep learning in recent years, hardware accelerators are continuously deployed in more and more safety-critical applications such as autonomous driving and robotics. While the accelerators are usually fabricated with advanced technology nodes for high performance and energy efficiency, they are also more prone to timing errors under process, voltage, temperature, and aging (PVTA) variations. By revisiting the physical sources of timing errors, we show that most of the timing errors in the accelerator are caused by a specific subset of input patterns, defined as critical input patterns. To improve the timing error resilience of the accelerator, in this paper, we propose READ, a reliability-enhanced acceler-tor dataflow optimization technique that can effectively reduce timing errors. READ reduces the occurrence of critical input patterns by exploring the optimal computing sequence when mapping a trained deep neural network to accelerators. READ only changes the order of multiply-accumulate operations in a convolution, which introduces negligible hardware overhead and no impact on accuracy. The experimental results on VGG and ResNet demonstrate on average$\mathbf{7.8}\times$timing error rate (TER) reduction and up to$\mathbf{37.9}\times$TER reduction for certain layers. The results also show that READ enables the accelerator to maintain accuracy over a wide range of PVTA variations, making it a promising approach for robust deep-learning design.

Memory-aware network scheduling is becoming increasingly important for deep neural network (DNN) inference on resource-constrained devices. However, due to the complex cell-level and network-level topologies, memory-aware scheduling becomes very challenging. While previous algorithms all suffer from poor scalability, in this paper, we propose an efficient memory-aware scheduling framework based on iterative computation graph optimization. Our framework features an iterative graph fusion algorithm that simplifies the computation graph while preserving the scheduling optimality. We further propose an integer linear programming formulation together with topology-aware variable pruning to schedule the simplified graph efficiently. We evaluate our method against prior-art algorithms on different networks and demonstrate that our method outperforms existing techniques in all the benchmarks, reducing the peak memory footprint by 13.4%, and achieving better scalability for networks with complex network-level topologies.

Secure multi-party computation (MPC) enables computation directly on encrypted data and protects both data and model privacy in deep learning inference. However, existing neural network architectures, including Vision Transformers (ViTs), are not designed or optimized for MPC and incur significant latency overhead. We observe Softmax accounts for the major latency bottleneck due to a high communication complexity, but can be selectively replaced or linearized without compromising the model accuracy. Hence, in this paper, we propose an MPC-friendly ViT, dubbed MPCViT, to enable accurate yet efficient ViT inference in MPC. Based on a systematic latency and accuracy evaluation of the Softmax attention and other attention variants, we propose a heterogeneous attention optimization space. We also develop a simple yet effective MPC-aware neural architecture search algorithm for fast Pareto optimization. To further boost the inference efficiency, we propose MPCViT+, to jointly optimize the Softmax attention and other network components, including GeLU, matrix multiplication, etc. With extensive experiments, we demonstrate that MPCViT achieves 1.9%, 1.3% and 3.6% higher accuracy with 6.2×, 2.9× and 1.9× latency reduction compared with baseline ViT, MPCFormer and THE-X on the Tiny-ImageNet dataset, respectively. MPCViT+further achieves a better Pareto front compared with MPCViT. The code and models for evaluation are available at https://github.com/PKU-SEC-Lab/mpcvit.

Deep neural network (DNN) inference based on secure 2-party computation (2PC) can offer cryptographically-secure privacy protection but suffers from orders of magnitude latency overhead due to enormous communication. Previous works heavily rely on a proxy metric of ReLU counts to approximate the communication overhead and focus on reducing the ReLUs to improve the communication efficiency. However, we observe these works achieve limited communication reduction for state-of-the-art (SOTA) 2PC protocols due to the ignorance of other linear and non-linear operations, which now contribute to the majority of communication. In this work, we present CoPriv, a framework that jointly optimizes the 2PC inference protocol and the DNN architecture. CoPriv features a new 2PC protocol for convolution based on Winograd transformation and develops DNN-aware optimization to significantly reduce the inference communication. CoPriv further develops a 2PC-aware network optimization algorithm that is compatible with the proposed protocol and simultaneously reduces the communication for all the linear and non-linear operations. We compare CoPriv with the SOTA 2PC protocol, CrypTFlow2, and demonstrate 2.1× communication reduction for both ResNet-18 and ResNet-32 on CIFAR-100. We also compare CoPriv with SOTA network optimization methods, including SNL, MetaPruning, etc. CoPriv achieves 9.98× and 3.88× online and total communication reduction with a higher accuracy compare to SNL, respectively. CoPriv also achieves 3.87× online communication reduction with more than 3% higher accuracy compared to MetaPruning.

As the timing guardband consumes more and more design margin with the technology scaling, better-than-worst-case (BTWC) techniques have gained more attention as a promising solution. BTWC techniques can relax the design margin by transcending the pessimistic static timing constraints and utilizing the dynamic timing information. However, to guarantee the design reliability throughout the lifetime, the conventional dynamic timing analysis (DTA) engines need an extra reliability guardband, which is commonly evaluated under the worst-case corners of aging and variation. This type of guardbanding consumes the precious design margin, thus hindering the efficiency improvement from BTWC techniques. Therefore, in this article, we propose AVATAR, an aging- and variation-aware dynamic timing analyzer that can perform DTA with the impact of transistor aging and random process variation, including the gate-level aging analysis and random variation model that can accurately calculate cell delay under the impact of transistor aging and random variation, and an event-based DTA algorithm that avoids the pessimistic property of graph-based analysis. We also propose a machine learning (ML)-assisted DTA acceleration flow for the multicycle DTA of homogeneous multicore designs. We present two case studies using AVATAR to show its effectiveness. First, we present an application-based dynamic-voltage-frequency-scaling (DVFS) design methodology based on AVATAR, which can exploit application-level dynamic timing slack (DTS) to improve energy efficiency and performance. The results demonstrate that, compared to the design based on the conventional corner-based DTA, the additional performance improvement of the design based on AVATAR can be up to 14% or the additional power-saving can be up to 20%. Second, we demonstrate using the proposed ML-assisted acceleration flow for reliability-aware deep neural network (DNN) accelerator simulation. We use the proposed flow to estimate the impact of timing errors due to aging and random variation on the inference accuracy of two benchmark DNNs. The results demonstrate that the proposed acceleration flow achieves up to$10\times $speedup with an average error of less than 2%.

We design deep neural networks (DNNs) and corresponding networks' splittings to distribute DNNs' workload to camera sensors and a centralized aggregator on head mounted devices to meet system performance targets in inference accuracy and latency under the given hardware resource constraints. To achieve an optimal balance among computation, communication, and performance, a split-aware neural architecture search framework, SplitNets, is introduced to conduct model designing, splitting, and communication reduction simultaneously. We further extend the framework to multi-view systems for learning to fuse inputs from multiple camera sensors with optimal performance and systemic efficiency. We validate SplitNets for single-view system on ImageNet as well as multi-view system on 3D classification, and show that the SplitNets framework achieves state-of-the-art (SOTA) performance and system latency compared with existing approaches.

Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to the convolutional neural network (CNN)-based models. However, ViTs mainly designed for image classification will generate single-scale low-resolution representations, which makes dense prediction tasks such as semantic segmentation challenging for ViTs. Therefore, we propose HRViT, which enhances ViTs to learn semantically-rich and spatially-precise multi-scale representations by integrating high-resolution multi-branch architectures with ViTs. We balance the model performance and efficiency of HRViT by various branch-block co-optimization techniques. Specifically, we explore heterogeneous branch designs, reduce the redundancy in linear layers, and augment the attention block with enhanced expressiveness. Those approaches enabled HRViT to push the Pareto frontier of performance and efficiency on semantic segmentation to a new level, as our evaluation results on ADE20K and Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction, demonstrating the potential of HRViT as a strong vision backbone for semantic segmentation. Our code is publicly available11https://github.com/facebookresearch/HRViT.

Contrastive learning learns visual representations by enforcing feature consistency under different augmented views. In this work, we explore contrastive learning from a new perspective. Interestingly, we find that quantization, when properly engineered, can enhance the effectiveness of contrastive learning. To this end, we propose a novel contrastive learning framework, dubbed Contrastive Quant, to encourage feature consistency under both differently augmented inputs via various data transformations and differently augmented weights/activations via various quantization levels. Extensive experiments, built on top of two state-of-the-art contrastive learning methods SimCLR and BYOL, show that Contrastive Quant consistently improves the learned visual representation.

From wearables to powerful smart devices, modern automatic speech recognition (ASR) models run on a variety of edge devices with different computational budgets. To navigate the Pareto front of model accuracy vs model size, researchers are trapped in a dilemma of optimizing model accuracy by training and fine-tuning models for each individual edge device while keeping the training GPU-hours tractable. In this paper, we propose Omni-sparsity DNN, where a single neural network can be pruned to generate optimized model for a large range of model sizes. We develop training strategies for Omni-sparsity DNN that allows it to find models along the Pareto front of word-error-rate (WER) vs model size while keeping the training GPU-hours to no more than that of training one singular model. We demonstrate the Omni-sparsity DNN with streaming E2E ASR models. Our results show great saving on training time and resources with similar or better accuracy on LibriSpeech compared to individually pruned sparse models: 2%-6.6% better WER on Test-other.

Efficient deep neural network (DNN) models equipped with compact operators (e.g., depthwise convolutions) have shown great potential in reducing DNNs’ theoretical complexity (e.g., the total number of weights/operations) while maintaining a decent model accuracy. However, existing efficient DNNs are still limited in fulfilling their promise in boosting real-hardware efficiency, due to their commonly adopted compact operators’ low hardware utilization. In this work, we open up a new compression paradigm for developing real-hardware efficient DNNs, leading to boosted hardware efficiency while maintaining model accuracy. Interestingly, we observe that while some DNN layers’ activation functions help DNNs’ training optimization and achievable accuracy, they can be properly removed after training without compromising the model accuracy. Inspired by this observation, we propose a framework dubbed DepthShrinker, which develops hardware-friendly compact networks via shrinking the basic building blocks of existing efficient DNNs that feature irregular computation patterns into dense ones with much improved hardware utilization and thus real-hardware efficiency. Excitingly, our DepthShrinker framework delivers hardware-friendly compact networks that outperform both state-of-the-art efficient DNNs and compression techniques, e.g., a 3.06% higher accuracy and 1.53x throughput on Tesla V100 over SOTA channel-wise pruning method MetaPruning. Our codes are available at: https://github.com/facebookresearch/DepthShrinker.

Collaborative learning enables distributed clients to learn a shared model for prediction while keeping the training data local on each client. However, existing collaborative learning methods require fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to the high labeling cost and the requirement of expertise. The lack of labels makes collaborative learning impractical in many realistic settings. Self-supervised learning can address this challenge by learning from unlabeled data. Contrastive learning (CL), a self-supervised learning approach, can effectively learn visual representations from unlabeled image data. However, the distributed data collected on clients are usually not independent and identically distributed (non-IID) among clients, and each client may only have few classes of data, which degrades the performance of CL and learned representations. To tackle this problem, we propose a collaborative contrastive learning framework consisting of two approaches: feature fusion and neighborhood matching, by which a unified feature space among clients is learned for better data representations. Feature fusion provides remote features as accurate contrastive information to each client for better local learning. Neighborhood matching further aligns each client’s local features to the remote features such that well-clustered features among clients can be learned. Extensive experiments show the effectiveness of the proposed framework. It outperforms other methods by 11% on IID data and matches the performance of centralized learning.

Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify a series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%. Code and models are available at:https://github.com/facebookresearch/bit.

Neural Architecture Search (NAS) automates and prospers the design of neural networks. Estimator-based NAS has been proposed recently to model the relationship between architectures and their performance to enable scalable and flexible search. However, existing estimator-based methods encode the architecture into a latent space without considering graph similarity. Ignoring graph similarity in node-based search space may induce a large inconsistency between similar graphs and their distance in the continuous encoding space, leading to inaccurate encoding representation and/or reduced representation capacity that can yield sub-optimal search results. To preserve graph correlation information in encoding, we propose NASGEM which stands for Neural Architecture Search via Graph Embedding Method. NASGEM is driven by a novel graph embedding method equipped with similarity measures to capture the graph topology information. By precisely estimating the graph distance and using an auxiliary Weisfeiler-Lehman kernel to guide the encoding, NASGEM can utilize additional structural information to get more accurate graph representation to improve the search efficiency. GEMNet, a set of networks discovered by NASGEM, consistently outperforms networks crafted by existing search methods in classification tasks, i.e., with 0.4%-3.6% higher accuracy while having 11%- 21% fewer Multiply-Accumulates. We further transfer GEMNet for COCO object detection. In both one-stage and twostage detectors, our GEMNet surpasses its manually-crafted and automatically-searched counterparts.

Neural network accelerator is a key enabler for the on-device AI inference, for which energy efficiency is an important metric. The datapath energy, including the computation energy and the data movement energy among the arithmetic units, claims a significant part of the total accelerator energy. By revisiting the basic physics of the arithmetic logic circuits, we show that the datapath energy is highly correlated with the bit flips when streaming the input operands into the arithmetic units, defined as the hamming distance (HD) of the input operand matrices. Based on the insight, we propose a post-training optimization algorithm and a HD-aware training algorithm to co-design and co-optimize the accelerator and the network synergistically. The experimental results based on post-layout simulation with MobileNetV2 demonstrate on average 2.85x datapath energy reduction and up to 8.51x datapath energy reduction for certain layers.

Data augmentation (DA) is an essential technique for training state-of-the-art deep learning systems. In this paper, we empirically show that the standard data augmentation methods may introduce distribution shift and consequently hurt the performance on unaugmented data during inference. To alleviate this issue, we propose a simple yet effective approach, dubbed KeepAugment, to increase the fidelity of augmented images. The idea is to use the saliency map to detect important regions on the original images and preserve these informative regions during augmentation. This information-preserving strategy allows us to generate more faithful training examples. Empirically, we demonstrate that our method significantly improves upon a number of prior art data augmentation schemes, e.g. AutoAugment, Cutout, random erasing, achieving promising results on image classification, semi-supervised image classification, multi-view multi-camera tracking and object detection.

Neural architecture search (NAS) has shown great promise in designing state-of-the-art (SOTA) models that are both accurate and efficient. Recently, two-stage NAS, e.g. BigNAS, decouples the model training and searching process and achieves remarkable search efficiency and accuracy. Two-stage NAS requires sampling from the search space during training, which directly impacts the accuracy of the final searched models. While uniform sampling has been widely used for its simplicity, it is agnostic of the model performance Pareto front, which is the main focus in the search process, and thus, misses opportunities to further improve the model accuracy. In this work, we propose AttentiveNAS that focuses on improving the sampling strategy to achieve better performance Pareto. We also propose algorithms to efficiently and effectively identify the networks on the Pareto during training. Without extra re-training or post-processing, we can simultaneously obtain a large number of networks across a wide range of FLOPs. Our discovered model family, AttentiveNAS models, achieves top-1 accuracy from 77.3% to 80.7% on ImageNet, and outperforms SOTA models, including BigNAS, Once-for-All networks and FBNetV3. We also achieve ImageNet accuracy of 80.1% with only 491 MFLOPs. Our training code and pretrained models are available at https://github.com/facebookresearch/AttentiveNAS.

Quantization is promising in enabling powerful yet complex deep neural networks (DNNs) to be deployed into resource constrained platforms. However, quantized DNNs are vulnerable to adversarial attacks unless being equipped with sophisticated techniques, leading to a dilemma of struggling between DNNs’ efficiency and robustness. In this work, we demonstrate a new perspective regarding quantization’s role in DNNs’ robustness, advocating that quantization can be leveraged to largely boost DNNs’ robustness, and propose a framework dubbed Double-Win Quant that can boost the robustness of quantized DNNs over their full precision counterparts by a large margin. Specifically, we for the first time identify that when an adversarially trained model is quantized to different precisions in a post-training manner, the associated adversarial attacks transfer poorly between different precisions. Leveraging this intriguing observation, we further develop Double-Win Quant integrating random precision inference and training to further reduce and utilize the poor adversarial transferability, enabling an aggressive “win-win" in terms of DNNs’ robustness and efficiency. Extensive experiments and ablation studies consistently validate Double-Win Quant’s effectiveness and advantages over state-of-the-art (SOTA) adversarial training methods across various attacks/models/datasets. Our codes are available at: https://github.com/RICE-EIC/Double-Win-Quant.

Weight-sharing neural architecture search (NAS) is an effective technique for automating efficient neural architecture design. Weight-sharing NAS builds a supernet that assembles all the architectures as its sub-networks and jointly trains the supernet with the sub-networks. The success of weight-sharing NAS heavily relies on distilling the knowledge of the supernet to the sub-networks. However, we find that the widely used distillation divergence, i.e., KL divergence, may lead to student sub-networks that over-estimate or under-estimate the uncertainty of the teacher supernet, leading to inferior performance of the sub-networks. In this work, we propose to improve the supernet training with a more generalized alpha-divergence. By adaptively selecting the alpha-divergence, we simultaneously prevent the over-estimation or under-estimation of the uncertainty of the teacher model. We apply the proposed alpha-divergence based supernets training to both slimmable neural networks and weight-sharing NAS, and demonstrate significant improvements. Specifically, our discovered model family, AlphaNet, outperforms prior-art models on a wide range of FLOPs regimes, including BigNAS, Once-for-All networks, and AttentiveNAS. We achieve ImageNet top-1 accuracy of 80.0% with only 444M FLOPs. Our code and pretrained models are available at https://github.com/facebookresearch/AlphaNet.

We present DIAN, a Differentiable Accelerator-Network Co-Search framework for automatically searching for matched networks and accelerators to maximize both the accuracy and efficiency. Specifically, DIAN integrates two enablers: (1) a generic design space for DNN accelerators that is applicable to both FPGA- and ASIC-based DNN accelerators; and (2) a joint DNN network and accelerator co-search algorithm that enables the simultaneous search for optimal DNN structures and their accelerators. Experiments and ablation studies based on FPGA measurements and ASIC synthesis show that the matched networks and accelerators generated by DIAN consistently outperform state-of-the-art (SOTA) DNNs and DNN accelerators (e.g., 3.04× better FPS with a 5.46% higher accuracy on ImageNet), while requiring notably reduced search time (up to $1234.3\times)$ over SOTA co-exploration methods, when evaluated over ten SOTA baselines on three datasets.

We introduce PyTorchVideo, an open-source deep-learning library that provides a rich set of modular, efficient, and reproducible components for a variety of video understanding tasks, including classification, detection, self-supervised learning, and low-level processing. The library covers a full stack of video understanding tools including multimodal data loading, transformations, and models that reproduce state-of-the-art performance. PyTorchVideo further supports hardware acceleration that enables real-time inference on mobile devices. The library is based on PyTorch and can be used by any training framework; for example, PyTorchLightning, PySlowFast, or Classy Vision. PyTorchVideo is available at https://pytorchvideo.org/.

As the integrated circuits (ICs) technology continues to scale, resolution enhancement techniques (RETs) are mandatory to obtain high manufacturing quality and yield. Among various RETs, subresolution assist feature (SRAF) generation is a key technique to improve the target pattern quality and lithographic process window. While model-based SRAF insertion techniques have demonstrated high accuracy, they usually suffer from high computational cost. Therefore, more efficient techniques that can achieve high accuracy while reducing runtime are in strong demand. In this article, we leverage the recent advancement in machine learning for image generation to tackle the SRAF insertion problem. In particular, we propose a new SRAF insertion framework, GAN-SRAF, which uses generative adversarial networks (GANs) to generate SRAFs directly for any given layout. Our proposed approach incorporates a novel layout to image encoding using multichannel heatmaps to preserve the layout information and facilitate layout reconstruction. Our experimental results demonstrate ~14.6× reduction in runtime when compared to the previous best machine learning approach for SRAF generation, and ~144× reduction compared to the model-based approach, while achieving comparable quality of results.

Neural Architecture Search (NAS) has demonstrated its power on various AI accelerating platforms such as Field Programmable Gate Arrays (FPGAs) and Graphic Processing Units (GPUs). However, it remains an open problem how to integrate NAS with Application-Specific Integrated Circuits (ASICs), despite them being the most powerful AI accelerating platforms. The major bottleneck comes from the large design freedom associated with ASIC designs. Moreover, with the consideration that multiple DNNs will run in parallel for different workloads with diverse layer operations and sizes, integrating heterogeneous ASIC sub-accelerators for distinct DNNs in one design can significantly boost performance, and at the same time further complicate the design space. To address these challenges, in this paper we build ASIC template set based on existing successful designs, described by their unique dataflows, so that the design space is significantly reduced. Based on the templates, we further propose a framework, namely ASICNAS, which can simultaneously identify multiple DNN architectures and the associated heterogeneous ASIC accelerator design, such that the design specifications (specs) can be satisfied, while the accuracy can be maximized. Experimental results show that compared with successive NAS and ASIC design optimizations which lead to design spec violations, ASICNAS can guarantee the results to meet the design specs with 17.77%, 2.49×, and 2.32× reductions on latency, energy, and area and less than 1.6% accuracy loss. To the best of the authors’ knowledge, this is the first work on neural architecture and ASIC accelerator design co-exploration.

Personalized recommendation systems leverage deep learning models and account for the majority of data center AI cycles. Their performance is dominated by memory-bound sparse embedding operations with unique irregular memory access patterns that pose a fundamental challenge to accelerate. This paper proposes a lightweight, commodity DRAM compliant, near-memory processing solution to accelerate personalized recommendation inference. The in-depth characterization of production-grade recommendation models shows that embedding operations with high model-, operator and data-level parallelism lead to memory bandwidth saturation, limiting recommendation inference performance. We propose RecNMP which provides a scalable solution to improve system throughput, supporting a broad range of sparse embedding models. RecNMP is specifically tailored to production environments with heavy co-location of operators on a single server. Several hardware/software cooptimization techniques such as memory-side caching, tableaware packet scheduling, and hot entry profiling are studied, providing up to 9.8× memory latency speedup over a highly-optimized baseline. Overall, RecNMP offers 4.2× throughput improvement and 45.8% memory energy savings.

With recent advances in reverse engineering, attackers can reconstruct a netlist to counterfeit chips by opening the die and scanning all layers of authentic chips. This relatively easy counterfeiting is made possible by the use of the standard simple clocking scheme, where all combinational blocks function within one clock period, so that a netlist of combinational logic gates and flip-flops is sufficient to duplicate a design. In this article, we propose to invalidate the assumption that a netlist completely represents the function of a circuit with unconventional timing. With the introduced wave-pipelining (WP) paths, attackers have to capture gate and interconnect delays during reverse engineering, or to test a huge number of combinational paths to identify the WP paths. To hinder the test-based attack, we construct false paths with WP to increase the counterfeiting challenge. The experimental results confirm that WP true paths and false paths can be constructed in benchmark circuits successfully with only a negligible cost, thus thwarting the potential attack techniques.

As modern integrated circuits scale up with escalating complexity of layout design patterns, lithography hotspot detection, a key stage of physical verification to ensure layout finishing and design closure, has raised a higher demand on its efficiency and accuracy. Among all the hotspot detection approaches, machine learning distinguishes itself for achieving high accuracy while maintaining low false alarms. However, due to the class imbalance problem, the conventional practice which uses the accuracy and false alarm metrics to evaluate different machine learning models is becoming less effective. In this work, we propose the use of the area under the ROC curve (AUC), which provides a more holistic measure for imbalanced datasets compared with the previous methods. To systematically handle class imbalance, we further propose the surrogate loss functions for direct AUC maximization as a substitute for the conventional cross-entropy loss. Experimental results demonstrate that the new surrogate loss functions are promising to outperform the cross-entropy loss when applied to the state-of-the-art neural network model for hotspot detection.

Optical neural network (ONN) is a neuromorphic computing hardware based on optical components. Since its first on-chip experimental demonstration, it has attracted more and more research interests due to the advantages of ultra-high speed inference with low power consumption. In this work, we design a novel slimmed architecture for realizing optical neural network considering both its software and hardware implementations. Different from the originally proposed ONN architecture based on singular value decomposition which results in two implementation-expensive unitary matrices, we show a more area-efficient architecture which uses a sparse tree network block, a single unitary block and a diagonal block for each neural network layer. In the experiments, we demonstrate that by leveraging the training engine, we are able to find a comparable accuracy to that of the previous architecture, which brings about the flexibility of using the slimmed implementation. The area cost in terms of the Mach-Zehnder interferometers, the core optical components of ONN, is 15%-38% less for various sizes of optical neural networks.

As the integrated circuits (IC) technology continues to scale, resolution enhancement techniques (RETs) are mandatory to obtain high manufacturing quality and yield. Among various RETs, sub-resolution assist feature (SRAF) generation is a key technique to improve the target pattern quality and lithographic process window. While model-based SRAF insertion techniques have demonstrated high accuracy, they usually suffer from high computational cost. Therefore, more efficient techniques that can achieve high accuracy while reducing runtime are in strong demand. In this work, we leverage the recent advancement in machine learning for image generation to tackle the SRAF insertion problem. In particular, we propose a new SRAF insertion framework, GAN-SRAF, which uses conditional generative adversarial networks (CGANs) to generate SRAFs directly for any given layout. Our proposed approach incorporates a novel layout to image encoding using multi-channel heatmaps to preserve the layout information and facilitate layout reconstruction. Our experimental results demonstrate ~14.6× reduction in runtime when compared to the previous best machine learning approach for SRAF generation, and ~144× reduction compared to model-based approach, while achieving comparable quality of results.

Logic locking and IC camouflaging are two promising techniques for thwarting an array of supply chain threats. Logic locking can hide the design from the foundry as well as end-users and IC camouflaging can thwart IC reverse engineering by end-users. Oracle-guided SAT-based deobfuscation attacks against these schemes have made it more and more difficult to securely implement them with low overhead. Almost all of the literature on SAT attacks is focused on combinational circuits. A recent first implementation of oracle-guided attacks on sequential circuits showed a drastic increase in deobfuscation time versus combinational circuits. In this paper we show that integrating the sequential SAT-attack with incremental bounded-model-checking, and dynamic simplification of key-conditions (Key-Condition Crunching or KC2), we are able to reduce the runtime of sequential SAT-attacks by two orders of magnitude across benchmark circuits, significantly reducing the gap between sequential and combinational deobfuscation. These techniques are applicable to combinational deobfuscation as well and thus represent a generic improvement to deobfuscation procedures and help better understand the complexity of deobfuscation for designing secure locking/camouflaging schemes.

Lithography hotspot detection is one of the fundamental steps in physical verification. Due to the increasingly complicated design patterns, early and quick feedback for lithography hotspots is desired to guide design closure in early stages. Machine learning approaches have been successfully applied to hotspot detection while demonstrating a remarkable capability of generalization to unseen hotspot patterns. However, most of the proposed machine learning approaches are not yet able to answer one critical question: how much a hotspot predicted from a trained model can be trusted? In this work, we present Litho-GPA, a lithography hotspot detection framework, with Gaussian Process assurance to provide confidence in each prediction. The framework also incorporates a data selection scheme with a sequence of weak classifiers to sample representative data and eventually reduce the amount of training data and lithography simulations needed. Experimental results demonstrate that our Litho-GPA is able to achieve the state-of-the-art accuracy while obtaining on average 28% reduction in false alarms.

The advancing of reverse engineering techniques has complicated the efforts in intellectual property protection. Proactive methods have been developed recently, among which layout-level integrated circuit camouflaging is the leading example. However, existing camouflaging methods are rarely supported by provably secure criteria, which further leads to an over-estimation of the security level when countering latest de-camouflaging attacks, e.g., the SAT-based attack. In this paper, a quantitative security criterion is proposed for de-camouflaging complexity measurements and formally analyzed through the demonstration of the equivalence between the existing de-camouflaging strategy and the active learning scheme. Supported by the new security criterion, two camouflaging techniques are proposed, including the low-overhead camouflaging cell generation strategy and the AND-tree camouflaging strategy, to help achieve exponentially increasing security levels at the cost of linearly increasing performance overhead on the circuit under protection. A provably secure camouflaging framework is then developed combining these two techniques. The experimental results using the security criterion show that camouflaged circuits with the proposed framework are of high resilience against different attack schemes with only negligible performance overhead.

Trojans and backdoors inserted by untrusted foundries have become serious threats to hardware security. Split manufacturing is proposed to hide important circuit structures and prevent Trojan insertion by fabricating partial interconnections in trusted foundries. Existing split manufacturing frameworks, however, usually lack security guarantee and suffer from poor scalability. It is observed that inserting dummy cells and wires can have high potential on overcoming the security and scalability problems of existing methods, but it is not compatible with current security definition. In this paper, we focus on answering the questions on how to define the notion of security and how to realize the required security level effectively and efficiently when the insertion of dummy cells and wires is considered. We first generalize existing security criterion by modeling the split manufacturing process as a graph problem. Then, a sufficient condition is derived for the proposed security criterion to avoid the computationally intensive operations in traditional methods. To further enhance the scalability of the framework, we propose a secure-by-construction split manufacturing flow. For the first time, a novel mixed-integer linear programming (MILP) formulation is proposed to simultaneously consider cell and wire insertion together with wire lifting. A Lagrangian relaxation algorithm with a minimum-cost flow transformation technique is employed to solve the MILP formulation efficiently. With extensive experiments, our framework demonstrates significantly better efficiency, overhead reduction and security guarantee compared with the previous state-of-the-art.

Lithography simulation is one of the key steps in physical verification, enabled by the substantial optical and resist models. A resist model bridges the aerial image simulation to printed patterns. While the effectiveness of learning-based solutions for resist modeling has been demonstrated, they are considerably data-demanding. Meanwhile, a set of manufactured data for a specific lithography configuration is only valid for the training of one single model, indicating low data efficiency. Due to the complexity of the manufacturing process, obtaining enough data for acceptable accuracy becomes very expensive in terms of both time and cost, especially during the evolution of technology generations when the design space is intensively explored. In this paper, we propose a new resist modeling framework for contact layers, utilizing existing data from old technology nodes and active selection of data in a target technology node, to reduce the amount of data required from the target lithography configuration. Our framework based on transfer learning and active learning techniques is effective within a competitive range of accuracy, i.e., $3 \times -10 \times $ reduction on the amount of training data with comparable accuracy to the state-of-the-art learning approach.

The SAT-based attacks are extremely successful in deobfuscating the traditional combinational logic locking and IC camouflaging schemes. While several SAT-resilient protection schemes that increase the minimum query count of the attack have been proposed recently, none of them satisfy the output corruptibility (error) criteria. Therefore, most of them were combined with high corruptibility schemes to achieve both corruptibility and high query count. These “compound” schemes are successful since existing SAT attacks are agnostic to the corruptibility of the protection scheme. In this paper, we propose an approximate SAT-based attack framework which focuses on the iterative convergence of an attack toward a better solution. This helps our attack reduce a compound scheme to a standalone SAT-resilient scheme. In addition, we relate the problem of minimum query count to a well-known graph problem, and we propose a novel technique to increase the corruptibility of SAT-resilient protection schemes in a controllable manner. This creates protection schemes that have both high query count and corruptibility. Furthermore, due to the approximation resiliency property of these schemes, approximate attacks provide no advantage over exact attacks when attacking them.

The globalization of the semiconductor supply chain introduces ever-increasing security and privacy risks. Two major concerns are IP theft through reverse engineering and malicious modification of the design. The latter concern in part relies on successful reverse engineering of the design as well. IC camouflaging and logic locking are two of the techniques under research that can thwart reverse engineering by end-users or foundries. However, developing low overhead locking/camouflaging schemes that can resist the ever-evolving state-of-the-art attacks has been a challenge for several years. This article provides a comprehensive review of the state of the art with respect to locking/camouflaging techniques. We start by defining a systematic threat model for these techniques and discuss how various real-world scenarios relate to each threat model. We then discuss the evolution of generic algorithmic attacks under each threat model eventually leading to the strongest existing attacks. The article then systematizes defences and along the way discusses attacks that are more specific to certain kinds of locking/camouflaging. The article then concludes by discussing open problems and future directions.

Layout-dependent effects (LDEs) are becoming increasingly important as technology node continues to shrink into the regime of FinFET transistors. Prior LDE studies mainly focus on accurate transistor modeling and fast circuit performance evaluations at the early lifetime of a design. Few studies have been performed on the layout dependency of the circuit aging towards the end of life (EOL). This study demonstrates that, due to transistor-level layout-dependent aging (LDA) behaviors, circuit-level timing degradations are greatly impacted by layout configurations, including length of diffusion and oxide spacing. In this paper, we propose the first circuit-level aging mitigation framework to improve the critical-path timing towards the EOL. Our framework features comprehensive LDA evaluations for standard cell timing, which shows that multiple-row height cells lead to worse EOL timing than singlerow height cells due to length-of-diffusion effects. We further propose a min-cost-flow-based placement approach to concurrently allocate the oxide spacing among neighboring standard cells, which generates much better EOL timing than a conventional greedy approach. Experimental results demonstrate that under the concurrent approach in the proposed aging mitigation framework, the total and worst negative slacks for EOL timing are on average reduced by 42% and 25%, respectively.

Trojans and backdoors inserted by untrusted foundries have become serious threats to hardware security. Split manufacturing is proposed to prevent Trojan insertion proactively. Existing methods depend on wire lifting to hide partial circuit interconnections, which usually suffer from large overhead and lack of security guarantee. In this paper, we propose a novel split manufacturing framework that not only guarantees to achieve the required security level but also allows for a drastic reduction of the introduced overhead. In our framework, insertion of dummy circuit cells and wires is considered simultaneously with wire lifting. To support cell and wire insertion, we propose a new security criterion, and further derive its sufficient condition to avoid computation intensive operations in traditional methods. Then, for the first time, a novel mixed integer linear programming formulation is proposed to simultaneously consider cell and wire insertion together with wire lifting, which significantly enlarges the design space to guarantee the realization of the sufficient condition under the security requirements and overhead constraints. With extensive experimental results, our framework demonstrates much better efficiency, overhead reduction, and security guarantee compared with existing methods.

Logic locking is an attractive defense against a series of hardware security threats. However, oracle guided attacks based on advanced Boolean reasoning engines such as SAT, ATPG and model-checking have made it difficult to securely lock chips with low overhead. While the majority of existing locking schemes focus on gate-level locking, in this paper we present a layout-inclusive interconnect locking scheme based on cross-bars of metal-to-metal programmable-via devices. We demonstrate how this enables configuring a large obfuscation key with a small number of physical key wires contributing to zero to little substrate area overhead. Dense interconnect locking based on these circuit level primitives shows orders of magnitude better SAT attack resiliency compared to an XOR/XNOR gate-insertion locking with the same key length which has a much higher overhead.

Lithography simulation is one of the key steps in physical verification, enabled by the substantial optical and resist models. A resist model bridges the aerial image simulation to printed patterns. While the effectiveness of learning-based solutions for resist modeling has been demonstrated, they are considerably data-demanding. Meanwhile, a set of manufactured data for a specific lithography configuration is only valid for the training of one single model, indicating low data efficiency. Due to the complexity of the manufacturing process, obtaining enough data for acceptable accuracy becomes very expensive in terms of both time and cost, especially during the evolution of technology generations when the design space is intensively explored. In this work, we propose a new resist modeling framework for contact layers that utilizes existing data from old technology nodes to reduce the amount of data required from a target lithography configuration. Our framework based on residual neural networks and transfer learning techniques is effective within a competitive range of accuracy, i.e., 2-10X reduction on the amount of training data with comparable accuracy to the state-of-the-art learning approach.

With the dramatic increase in the complexity of modern integrated circuits (ICs), direct analysis and verification of IC power distribution networks (PDNs) have become extremely computationally expensive. Various power grid reduction methods are proposed to reduce the grid size for fast verification and simulation but usually suffer from poor scalability. In this paper, we present a convex optimization-based framework for power grid reduction. Edge sparsification is formulated as a weighted convex optimization problem with sparsity-inducing penalties, which provides an accurate control over the final error. A greedy coordinate descent (GCD) method with optimality guarantee is proposed along with a novel coordinate selection strategy to improve the efficiency and accuracy of edge sparsification. Experimental results demonstrate that the proposed approach achieves better performance compared with traditional gradient descent methods, and 98% accuracy and good sparsity for industrial benchmarks.

In order to counter advanced reverse engineering techniques, various integrated circuit (IC) camouflaging methods are proposed to protect hardware intellectual property (IP) proactively. For example, a timing-based camouflaging strategy is developed recently representing a new class of parametric camouflaging strategies. Unlike traditional IC camouflaging techniques that directly hide the circuit functionality, the new parametric strategies obfuscate the circuit timing schemes, which in turn protects the circuit functionality and invalidates all the existing attacks. In this paper, we propose a SAT attack, named TimingSAT, to analyze the security of such timing-based camouflaging strategies. We demonstrate that with a proper transformation of the camouflaged netlist, traditional SAT attacks are still effective to decamouflage the new protection methods. The correctness of the resolved circuit functionality is formally proved. While a direct implementation of TimingSAT suffers from poor scalability, we propose a simplification procedure to significantly enhance the attack efficiency without sacrificing the correctness of the decamouflaged netlist. The efficiency and effectiveness of TimingSAT is validated with extensive experimental results.

Quantum computing has raised great interests for its potential to achieve an asymptotic speedup on specific problems. Current quantum devices suffer from noise which needs robust and scalable error-correcting schemes. Topological quantum error correction (TQEC) is among the most promising error-correcting techniques with exponential suppression of error with linear increase of space-time complexity. In this paper, we present the first work to explore space-time optimization between 1-D and 2-D architectures for TQEC circuits. We prove the NP-hardness of the qubit routing problem in the layout synthesis and propose an efficient algorithm to optimize space-time volumes for both 1-D and 2-D qubit architectures with promising experimental results.

Digital circuit physical unclonable function (PUF) has been attracting attentions for the merits of resilience to the environmental and operational variations that analog PUFs suffer from. Existing state-of-the-art digital circuit PUFs, however, are either hybrid of analog-digital circuits which are still under the shadow of vulnerability, or impractical for real-world applications. In this paper, we propose a novel highly nonlinear and secure digital PUF (D-PUF) and the spliced version SD-PUF. The fingerprints are extracted from intentionally induced very large-scale integration interconnect randomness during lithography process, as well as a post-silicon shuffling process. Strongly skewed CMOS latches are used to ensure the immunity against environmental and operational variations. Crucially, a highly nonlinear logic network is proposed to effectively spread and augment any subtle interconnect randomness, which also enables strong resilience against machine learning attacks. On top of it, the expandable architecture of the proposed logic network empowers a novel post-silicon shuffle-splice mechanism, where multiple randomly selected D-PUFs are spliced to be one SD-PUF, pushing the statistical security to a much higher level, while significantly reducing the mask cost per PUF device. It also decouples the trustworthy demands enforced to the foundries or other third party manufacturers. Our proposed PUFs demonstrate close to ideal performance in terms of statistical metrics, including 0 intra-Hamming distance. Various state-of-the-art machine learning models show prediction accuracies almost no better than random guesses when attacking to the proposed PUFs. We also mathematically prove the probability of existence of identical SD-PUF pair is significantly lower than that of D-PUF pair, e.g., such probability of an SD-PUF spliced by 30 D-PUFs is 2.3 × 10-22, which is 19 order magnitude lower than that of D-PUF. Benefited from the proposed shuffle-splice mechanism, the mask cost per SD-PUF is also reduced by 300× than that of D-PUF.

Subresolution assist feature (SRAF) generation is a very important resolution enhancement technique to improve yield in modern semiconductor manufacturing process. Model-based and rule-based approaches are widely adopted in the semiconductor industry. The model-based SRAF generation can achieve a high accuracy but it is known to be time-consuming and it is hard to obtain consistent SRAFs on the same layout pattern configurations. The rule-based SRAF generation is highly technology dependent and it is becoming extremely difficult to render high-quality results in advanced technology nodes. This paper proposes supervised data learning techniques for fast yet consistent SRAF generation with high-quality results. We first propose the constrained concentric circle with area sampling scheme for feature extraction. Illumination source symmetry-based feature compaction technique is further invented to reduce the training data set size and achieve consistent SRAF predictions. Using accurate model-based SRAFs as training data, classification models based on logistic regression (LGR) and support vector machine are calibrated for SRAF predictions. Moreover, the probability maximum prediction is proposed to generate manufacturing-friendly SRAFs with a greedy simplification scheme. We compare support vector machine and LGR models by embedding into an entire mask optimization flow, where the support vector machine model obtains better lithographic performance. Experimental results demonstrate that, compared with the commercial Calibre tool, supervised data learning techniques for SRAF generation obtain significant speed up (>3X for a 100 um2layout clip) and comparable lithographic performance in terms of edge placement error and process variation band.

Modern field-programmable gate array (FPGA) devices contain complex clock architectures on top of configurable logics. Unlike application specific integrated circuits (ASICs), the physical structure of clock networks in an FPGA is pre-manufactured and cannot be adjusted to different applications. Furthermore, clock routing resources are typically limited for high-utilization designs. Consequently, clock architectures impose extra clock constraints and further complicate physical implementation tasks such as placement. Traditional ASIC placement techniques only optimize conventional design metrics such as wirelength, routability, power, and timing without clock legality consideration. It is imperative to have new techniques to honor clock constraints during placement for FPGAs. In this article, we propose a high-performance FPGA placement engine, UTPlaceF 2.0, that optimizes wirelength and routability while honoring complex clock constraints. Our proposed approaches consist of an iterative minimum-cost-flow-based cell assignment as well as a clock-aware packing for producing clock-legal yet high-quality placement solutions. UTPlaceF 2.0 won first place in the ISPD’17 clock-aware FPGA placement contest organized by Xilinx, outperforming the second- and the third-place winners by 4.0% and 10.0%, respectively, in routed wirelength with competitive runtime, on a set of industry benchmarks.

Fault attack becomes a serious threat to system security and requires to be evaluated in the design stage. Existing methods usually ignore the intrinsic uncertainty in attack process and suffer from low scalability. In this paper, we develop a general framework to evaluate system vulnerability against fault attack. A holistic model for fault injection is incorporated to capture the probabilistic nature of attack process. Based on the probabilistic model, a security metric named as System Security Factor (SSF) is defined to measure the system vulnerability. In the framework, a Monte Carlo method is leveraged to enable a feasible evaluation of SSF for different systems, security policies, and attack techniques. We enhance the framework with a novel system pre-characterization procedure, based on which an importance sampling strategy is proposed. Experimental results on a commercial processor demonstrate that compared to random sampling, a 2500X speedup is achieved with the proposed sampling strategy. Meanwhile, 3% registers are identified to contribute to more than 95% SSF. By hardening these registers, a 6.5X security improvement can be achieved with less than 2% area overhead.

Logic locking and IC camouflaging are proactive circuit obfuscation methods that if proven secure can thwart hardware attacks such as reverse engineering and IP theft. However, the security of both these schemes is called into question by recent SAT based attacks. While a number of methods have been proposed in literature that exponentially increase the running time of such attacks, they are vulnerable to "findand-remove" attacks, and only slightly hide the circuit functionality. In this paper, we present a novel approach towards creating SAT attack resiliency based on creating densely cyclic obfuscated circuit topologies by adding dummy paths to the circuit. Our methodology is applicable to both IC camouflaging and logic locking. We demonstrate that cyclic logic locking creates SAT resilient circuits with 40% less area and 20% less delay compared to an insecure XOR/XNOR-obfuscation with the same key length. Furthermore, we show that cyclic IC camouflaging can be implemented at the layout level with no substrate area overhead and little delay and power overhead with respect to the original circuit.

This paper provides a systematization of knowledge in the domain of integrated circuit protection through obfuscation with a focus on the recent Boolean satisfiability (SAT) attacks. The study systematically combines real-world IC reverse engineering reports, experimental results using the most recent oracle-guided attacks, and concepts in machine-learning and cryptography to draw a map of the state-of-the-art of IC obfuscation and future challenges and opportunities.

Global placement is a major runtime bottleneck of modern FPGA physical synthesis. As the FPGA capacity grows rapidly, new innovative global placement approaches are in great demand for more efficient circuit mapping and prototyping. In this paper, we propose a parallelization framework for modern FPGA global placement, UTPlaceF 3.0. Two major techniques are presented to boost the performance of a state-of-the-art quadratic placer with only small quality degradation: 1) placement-driven block-Jacobi preconditioning and 2) parallelized incremental placement correction. Experimental results show that UTPlaceF 3.0 can take full advantages of modern multi-core CPUs and achieves more than 5X speedup over sequential implementation with competitive placement quality.

As the semiconductor manufacturing technology continues to scale down to sub-10 nm, unidirectional layout style has become the mainstream for lower metal layers with tight pitches. Conventional redundant via (RV) insertion for yield improvement has become obsolete because unidirectional routing patterns forbid off-track routing, i.e., wire bending, for the metal coverage of RVs. To enhance the yield, redundant local-loop insertion (RLLI) is a new way of inserting RVs due to its compatibility with the unidirectional layout style. This paper proposes the first global optimization engine for RLLI considering advanced manufacturing constraints. Our key contributions include bounded timing impact analysis and evaluation for the local-loop structure, net-based local-loop candidate generation and pruning, an integer linear programming (ILP) formulation and scalable iterative relaxation/linear programming solving (IRLS) with incremental search scheme. Experimental results demonstrate that with bounded timing impact (within 1%), the ILP formulation obtains highest insertion rate while the IRLS with incremental search scheme achieves scalable solutions with competitive solution qualities.

The execution-simulation gap (ESG) is a fundamental property of public physical unclonable function (PPUF), which exploits the time gap between direct IC execution and computer simulation. ESG needs to consider both advanced computing scheme, including parallel and approximate computing scheme, and IC physical realization. In this paper, we propose a novel PPUF design, whose execution is equivalent to solving the hard-to-parallel and hard-to-approximate max-flow problem in a complete graph on chip. Thus, max-flow problem can be used as the simulation model to bound the ESG rigorously. To enable an efficient physical realization, we propose a crossbar structure and adopt source degeneration technique to map the graph topology on chip. The difference on asymptotic scaling between execution delay and simulation time is examined in the experimental results. The measurability of output difference is also verified to prove the physical practicality.

An efficient methodology for soft error analysis of sequential circuits based on Monte Carlo sampling is proposed. It uses nested sampling for faster statistical convergence: it samples only from the workload space and statically evaluates the conditional probability over the subspace of particle strike and circuit parameters. A novel check on the stationarity of machine state sequence to reduce the number of samples to convergence is introduced. The flow combines logic simulation for latch-level error propagation and stationarity diagnostic and an improved combinational error simulator with a new masking model based on signal controllability.

The advancing of reverse engineering techniques has complicated the efforts in intellectual property protection. Proactive methods have been developed recently, among which layout-level IC camouflaging is the leading example. However, existing camouflaging methods are rarely supported by provably secure criteria, which further leads to over-estimation of the security level when countering the latest de-camouflaging attacks, e.g., the SAT-based attack. In this paper, a quantitative security criterion is proposed for de-camouflaging complexity measurements and formally analyzed through the demonstration of the equivalence between the existing de-camouflaging strategy and the active learning scheme. Supported by the new security criterion, two novel camouflaging techniques are proposed, the low-overhead camouflaging cell library and the AND-tree structure, to help achieve exponentially increasing security levels at the cost of linearly increasing performance overhead on the circuit under protection. A provably secure camouflaging framework is then developed by combining these two techniques. Experimental results using the security criterion show that the camouflaged circuits with the proposed framework are of high resilience against the SAT-based attack with negligible performance overhead.

Conventional silicon physical unclonable function (PUF) extracts fingerprints from transistor's analog attributes, which are vulnerable to environmental and operational variations. Recently, digitalized PUF prototypes have emerged to overcome the vulnerability issues, however, the existing prototypes are either hybrid of analog-digital PUFs which are still under the shadow of vulnerability, or impractical for real-world implementation. To address the above limitations, we propose a learning resilient and reliable digital PUF (LRR-DPUF). The fingerprints are extracted from VLSI interconnect geometrical randomness induced by lithography variations. Crucially, we use strongly skewed latches to ensure the immunity against environmental and operational variations. Further, a cross-coupled, highly non-linear logic network is proposed to effectively spread and augment even subtle interconnect randomness, as well as to achieve strong resilience to machine learning attacks. We demonstrate that a 64-bit LRR-DPUF exhibits close to ideal statistical performances, including 0 intra Hamming Distance. We also mathematically prove that each output of the LRR-DPUF follows uniform distribution. Various state-of-the-art machine learning models show almost no better than random prediction accuracies when applied to LRR-DPUF.

Model order reduction exploiting the spectral properties of the admittance matrix, known as the graph Laplacian, to control the approximation accuracy is a promising new class of approaches to power grid analysis. In this paper we introduce a method that allows a dramatic increase in the resulting graph sparsity and can handle large dense input graphs. The method is based on the observation that the information about the realistic ranges of port currents can be used to significantly improve the resulting graph sparsity. In practice, port currents cannot vary unboundedly and the estimates of peak currents are often available early in the design cycle. However, the existing methods including the sampling-based spectral sparsification approach [11] cannot utilize this information.