Hyperdimensional computing (HDC) is an emerging bio-inspired machine learning scheme for its balance between accuracy and efficiency. Similar to other machine learning models, a trained HDC model is a valuable intellectual property (IP) and subject to model stealing attacks. In this paper, we propose StealHD to functionally steal the HDC model under the strict and challenging ''data-free and black-box'' scenario, where the attacker can only query input to HDC models and observe corresponding output without knowing what data are used to train the model or the parameters in the model. Inspired by the characteristics of HDC algorithm, we leverage three noise back-ends and eight different augmentations to generate a surrogate dataset for attack following those patterns. We then feed the input to the victim model as queries to obtain the output similarities, based on which we train a surrogate HDC model to mimic the performance of the victim model. Experimental results on three datasets show that StealHD can effectively steal an HDC model under a strict data-free black-box scenario.

Modern deep learning-based recommendation system (DRS) is a dominant workload in industrial data centers. However, with continuous transistor scaling and increasing hardware complexity, silent data corruption (SDC) has become a notable threat to the reliability of data center workloads. Multiple industry hyperscalars have reported the difficulty in addressing SDC due to their “stealthy” nature and elusive manifestation. Given the critical role that DRS plays in maintaining the quality of online services, understanding and enhancing its robustness against SDCs are imperative. To the best our knowledge, this paper presents the first empirical study on understanding DRS robustness against SDCs. Specifically, we develop PyTEI, a PyTorch-based, user-friendly, and highly-efficient error injection framework, based on which we perform large-scale error injection experiments to the parameters of five representative DRS models under three datasets. Experimental results reveal that, the sparsity level of input data and feature affect the robustness of DRS, and in particular, MLP modules inside DRS are especially vulnerable to SDCs. Further, we evaluate the effectiveness of three representative error mitigation methods – algorithm based fault tolerance (ABFT), activation clipping, and selective bit protection (SBP), in enhancing DRS robustness. Experimental results reveal that activation clipping obtains the best result by recovering up to 30% of the degraded DRS performance under SDCs. This study provides valuable insights for industry practitioners in developing robust fault-tolerant strategies for DRS workloads. We open source PyTEI at https://github.com/facebookresearch/PyTEI.

Deep neural networks (DNNs) have been widely-adopted in various safety-critical applications such as computer vision and autonomous driving. However, as technology scales and applications diversify, coupled with the increasing heterogeneity of underlying hardware architectures, silent data corruption (SDC) has been emerging as a pronouncing threat to the reliability of DNNs. Recent reports from industry hyperscalars underscore the difficulty in addressing SDC due to their "stealthy" nature and elusive manifestation. In this paper, we propose Dr. DNA, a novel approach to enhance the reliability of DNN systems by detecting and mitigating SDCs. Specifically, we formulate and extract a set of unique SDC signatures from the Distribution of Neuron Activations (DNA), based on which we propose early-stage detection and mitigation of SDCs during DNN inference. We perform an extensive evaluation across 3 vision tasks, 5 different datasets, and 10 different models, under 4 different error models. Results show that Dr. DNA achieves 100% SDC detection rate for most cases, 95% detection rate on average and >90% detection rate across all cases, representing 20% - 70% improvement over baselines. Dr. DNA can also mitigate the impact of SDCs by effectively recovering DNN model performance with <1% memory overhead and <2.5% latency overhead.

Embedding tables in deep recommender systems (DRS) process categorical data, which can be memory-intensive due to the high feature cardinality. In this paper, we propose Approximate Rotary Compositional Embedding (ARCE), which intentionally trades off performance to aggressively reduce the size of the embedding tables. Specifically, ARCE uses compositional embedding to split large embedding tables into smaller compositions and replaces index look-ups with vector rotations. To regain the performance loss of this trade-off, ARCE features an input approximation where one index is mapped into multiple indices, creating a larger space for a potential increased learning capability. Experimental results show that using ARCE can reduce the memory overhead of embedding tables in DRS by more than 1000x with less than 3% performance loss, highlighting the potential of using ARCE for less memory intensive DRS designs. We open-source ARCE at https://github.com/VU-DETAIL/arce.

Hyperdimensional Computing (HDC), also known as Vector Symbolic Architecture (VSA), is an emerging AI algorithm inspired by the way the human brain functions. Compared with deep neural networks (DNNs), HDC possesses several advantages such as smaller model size, less computation cost, and one/few-shot learning, making it a promising alternative computing paradigm. With the increasing deployment of AI in safety-critical systems such as healthcare and robotics, it is not only important to strive for high accuracy, but also to ensure its robustness under even highly uncertain and adversarial environments. However, recent studies show that HDC, just like DNNs, is vulnerable to both cyber attacks (e.g., adversarial attacks) and hardware errors (e.g., memory failures). While a growing body of research has been studying the robustness of HDC, there is a lack of systematic review of research efforts on this increasingly-important topic. To the best of our knowledge, this paper presents the first survey dedicated to review the research efforts made to the robustness of HDC against cyber attacks and hardware errors. While the performance and accuracy of HDC as an AI method still expects future theoretical advancement, this survey paper aims to shed light and call for community efforts on robustness research of HDC.

Federated learning is a decentralized machine learning strategy that trains the model by using data stored across multiple decentralized edge devices or servers. Studies on federated learning currently focus primarily on neural network-based learning methods, which usually require powerful hardware and are relatively not energy-efficient. Recently, hyperdimensional computing (HDC) emerges as a potential alternative solution to neural networks, particularly on resource-constrained platforms such as edge intelligence systems. HDC mimics the “human brain” at the functionality level that learns with the attributes of brain circuits, including high-dimensionality and fully distributed holographic representation. Although there are existing works related to HDC-based federated learning, a comprehensive study on how HDC-based federated learning performs in different settings is still absent. To bridge this gap, we present a comprehensive case study on federated learning using HDC under two model aggregation strategies: hypervector aggregation and associative memory aggregation. We also perform extensive experiments with various settings, including data distribution, number of clients, and local training epochs. We also analyze their communication costs under these settings. Our results show that using the strategy of associative memory aggregation can achieve up to 95% communication cost reduction compared to hypervector aggregation. In addition, HDC-based federated learning system shows high robustness in training with Non-IID data. This study aims to shed light and provide guidance in opening up new directions and challenges for future HDC-based federated learning system design and optimization.

Brain-inspired hyperdimensional computing (HDC), also known as vector symbolic architecture (VSA), is an emerging “non-von Neumann” computing scheme that imitates human brain functions to process information or perform learning tasks using abstract and high-dimensional patterns. Compared with deep neural networks (DNNs), HDC shows advantages, such as compact model size, energy efficiency, and few-shot learning. Despite of those advantages, one under-investigated area of HDC is the adversarial robustness; existing works have shown that HDC is vulnerable to adversarial attacks where attackers can add minor perturbations onto the original inputs to “fool” HDC models, producing wrong predictions. In this article, we systematically study the adversarial robustness of HDC by developing a systematic approach to test and enhance the robustness of HDC against adversarial attacks with two main components: 1) TestHD, which is a highly automated testing tool that can generate high-quality adversarial data for a given HDC model and 2) GuardHD, which utilizes the adversarial data generated by TestHD to enhance the adversarial robustness of HDC models. The core idea of TestHD is built on top of fuzz testing method. We customize the fuzzing approach by proposing a similarity-based coverage metric to guide TestHD to continuously mutate original inputs to generate new inputs that can trigger incorrect behaviors of HDC model. Thanks to the use of differential testing, TestHD does not require knowing the labels of the samples beforehand. For enhancing the adversarial robustness, we design, implement, and evaluate GuardHD to defend HDC models against adversarial data. The core idea of GuardHD is an adversarial detector which can be trained by TestHD-generated adversarial samples. During inference, once an adversarial sample is detected, GuardHD will override the prediction result with an “invalid” signal. We evaluate the proposed methods on four datasets and five adversarial attack scenarios with six adversarial generation strategies and two defense mechanisms, and compare the performance correspondingly. GuardHD is able to differentiate between benign and adversarial inputs with over 90% accuracy, which is up to 55% higher than adversarial training-based baselines. To the best of our knowledge, this article presents the first comprehensive effort in systematically testing and enhancing the robustness against adversarial data of this emerging brain-inspired computational model.

In this paper, we propose MoleHD, an efficient learning model based on brain-inspired hyperdimensional computing (HDC) for molecular property prediction. We develop HDC encoders to project SMILES representation of a molecule into high-dimensional vectors that are used for HDC training and inference. We perform an extensive evaluation using 29 classification tasks from 3 widely-used molecule datasets (Clintox, BBBP, SIDER) under three splits methods (random, scaffold, and stratified). By a comprehensive comparison with 8 existing learning models, we show that MoleHD achieves highest ROC-AUC score on random and scaffold splits on average across 3 datasets and achieve second-highest on stratified split. More importantly, MoleHD achieves such performance with significantly reduced computing cost: no back-propagation needed, only around 10 minutes training time using CPU.MoleHD is open-sourced and available at https://github.com/VU-DETAIL/MoleHD.

Hyperdimensional Computing (HDC) is rapidly emerging as an attractive alternative to traditional deep learning algorithms. Despite the profound success of Deep Neural Networks (DNNs) in many domains, the amount of computational power and storage that they demand during training makes deploying them in edge devices very challenging if not infeasible. This, in turn, inevitably necessitates streaming the data from the edge to the cloud which raises serious concerns when it comes to availability, scalability, security, and privacy. Further, the nature of data that edge devices often receive from sensors is inherently noisy. However, DNN algorithms are very sensitive to noise, which makes accomplishing the required learning tasks with high accuracy immensely difficult. In this paper, we aim at providing a comprehensive overview of the latest advances in HDC. HDC aims at realizing real-time performance and robustness through using strategies that more closely model the human brain. HDC is, in fact, motivated by the observation that the human brain operates on high-dimensional data representations. In HDC, objects are thereby encoded with high-dimensional vectors which have thousands of elements. In this paper, we will discuss the promising robustness of HDC algorithms against noise along with the ability to learn from little data. Further, we will present the outstanding synergy between HDC and beyond von Neumann architectures and how HDC opens doors for efficient learning at the edge due to the ultra-lightweight implementation that it needs, contrary to traditional DNNs.

Recently, brain-inspired hyperdimensional computing (HDC) has demonstrated promising capability in a wide range of applications such as medical diagnosis, human activity recognition, and voice classification, etc. Despite the growing popularity of HDC, its memory-centric computing characteristics make the associative memory implementation under significant energy consumption due to the massive data storage and processing. In this paper, we present a systematic case study to leverage the application-level error resilience of HDC to reduce the energy consumption of HDC associative memory by using voltage scaling. Evaluation results on various applications show that our proposed approach can achieve 47.6% energy saving on associative memory with a 1% accuracy loss. We further explore two low-cost error masking methods: word masking and bit masking, to mitigate the impact of voltage scaling-induced errors. Experimental results show that the proposed word masking (bit masking) method can further enhance energy saving up to 62.3% (72.5%) with accuracy loss ≤1%.

Timing errors of microelectronic circuits occur when the circuit timing specification is violated, i.e., the dynamic delay of circuits exceeds the circuit clock period. With the continuous scaling of CMOS technology, microelectronic circuits are increasingly susceptible to microelectronic variations such as variations in operating conditions. Such variations can cause delay uncertainty in microelectronic circuits, leading totiming errors. Circuit designers typically combat these errors using conservative guardbands in the circuit and architectural design, which can, however, cause significant loss of operational efficiency. In this article, we proposeDEVoT, a supervised learning model that can predict the dynamic delay of functional units (FUs) under different operating conditions, clock speeds, and input workload. The main contribution ofDEVoTis to jointly consider the impact of voltage, temperature, and input workload in path sensitization, hence predicting the dynamic delay. We measure the dynamic delay using switching activity generated through gate-level simulation of post place-and-route design in the TSMC 45-nm process. We characterize the delay of FUs under different operating conditions and input workload. We then extract useful features in the input workload that influences dynamic path sensitization. Using these features, we apply supervised learning methods to buildDEVoT. Across 100 different operating conditions, four widely used FUs, and three datasets,DEVoTachieves, on average, less than 2% relative deviation from the ground truth and is$100\times $faster than the gate-level simulation. We present two case studies usingDEVoT. First, we useDEVoTto predict timing errors of FUs, andDEVoTachieves an average prediction accuracy at 98.04%. We further useDEVoTto estimate application output quality under different operating conditions, andDEVoTachieves an average estimation accuracy at 97% for two image processing applications. Second, we present a fuzzing-based method to identify “critical” patterns that can cause longer delay for a given circuit. Built on top ofDEVoT, the generated input patterns can improve the sensitized delay by up to 8.3% compared to random patterns.DEVoTalso outperforms automatic test pattern generation (ATPG) in sensitizing circuit delay. We will opensourceDEVoT, which can assist circuit designers to perform early design space exploration and can also help software developers in approximate computing community to assess their program resilience to hardware approximation without performing circuit simulation.

Brain-inspired hyperdimensional computing (HDC) is an emerging computational paradigm that mimics brain cognition and leverages hyperdimensional vectors with fully distributed holographic representation and (pseudo)randomness. Compared to other machine learning (ML) methods such as deep neural networks (DNNs), HDC offers several advantages including high energy efficiency, low latency, and one-shot learning, making it a promising alternative candidate on a wide range of applications. However, the reliability and robustness of HDC models have not been explored yet. In this paper, we design, implement, and evaluate HDTest to test HDC model by automatically exposing unexpected or incorrect behaviors under rare inputs. The core idea of HDTest is based on guided differential fuzz testing. Guided by the distance between query hypervector and reference hypervector in HDC, HDTest continuously mutates original inputs to generate new inputs that can trigger incorrect behaviors of HDC model. Compared to traditional ML testing methods, HDTest does not need to manually label the original input. Using handwritten digit classification as an example, we show that HDTest can generate thousands of adversarial inputs with negligible perturbations that can successfully fool HDC models. On average, HDTest can generate around 400 adversarial inputs within one minute running on a commodity computer. Finally, by using the HDTest-generated inputs to retrain HDC models, we can strengthen the robustness of HDC models. To the best of our knowledge, this paper presents the first effort in systematically testing this emerging brain-inspired computational model.

Approximate computing recently arises due to its success in many error-tolerant applications such as multimedia applications. Various approximation methods have demonstrated the effectiveness of relaxing precision requirements in a specific arithmetic unit. This provides a basis for exploring simultaneous use of multiple approximate units to improve efficiency. In this paper, we aim to identify a proper approximation configuration of approximate units in a program to minimize energy consumption while meeting quality constraints. To do this, we formulate a constrained optimization problem and develop a tool called WOAxC that uses genetic algorithm to solve this problem. WOAxC considers the impact of different input workload on the application quality. We evaluate the efficacy of WOAxC in minimizing the energy consumption of several image processing applications with varying size (i.e., number of operations), workload (i.e., input datasets), and quality constraints. Our evaluation shows that the configuration provided by WOAxC for a system with multiple approximate units improves the energy efficiency by, on average, 79.6%, 77.4%, and 70.94% for quality loss of 5%, 2.5% and 0% (no loss), respectively. To the best of our knowledge, WOAxC is the first workload-aware approach to identify proper approximation configuration for energy minimization under quality guarantee.

With the continuous scaling of CMOS technology, microelectronic circuits are increasingly susceptible to micro-electronic variations such as variations in operating conditions. Such variations can cause delay uncertainty in microelectronic circuits, leading to timing errors. Circuit designers typically combat these errors using conservative guardbands in the circuit and architectural design, which can, however, cause significant loss of operational efficiency. In this paper, we propose TEVoT, a supervised learning model that can predict the timing errors of functional units (FUs) under different operating conditions, clock speeds, and input workload. We perform dynamic timing analysis to characterize the delay variations of FUs under different conditions, based on which we collect training data. We then extract useful features from training data and apply supervised learning methods to establish TEVoT. Across 100 different operating conditions, 4 widely-used FUs, 3 clocking speeds, and 3 datasets, TEVoT achieves an average prediction accuracy at 98.25% and is 100X faster than gate-level simulation. We further use TEVoT to estimate application output quality under different operating conditions by exposing circuit-level timing errors to application level. TEVoT achieves an average estimation accuracy at 97% for two image processing applications across 100 operating conditions.

Recent years have witnessed a rapid growth of data-intensive applications such as machine learning and multimedia applications. However, such applications incur a heavy computation workload that stresses the existing computing systems, especially resource-constrained embedded systems. This paper is inspired by the key observation that many data-intensive applications naturally present a strong existence of trivial computations - a set of computations the results of which can be determined without actual computations. Typical examples include multiplication with 0, +1/-1 and addition with 0. Correspondingly, we develop and implement bypass circuits that are tightly integrated with computation units to detect and bypass the trivial computations. Once detected, the circuit delivers the pre-determined result without an actual computation. We implement bypass circuits in both hardware (Verilog) and software (C). Furthermore, we enhance the opportunities of computation bypass by developing AxBy, an approximate computation bypass method with pattern matching under limited data precision. This reconfigurability is key to achieving a “controllable approximation” and a tunable quality-energy tradeoff. Our experimental results show that for four image processing applications and three neural network applications, the computation bypass can enable 15% - 55% in image processing and 30% - 35% in neural networks of energy saving without any accuracy loss. For neural networks, we can further achieve 36% -44% energy saving with negligible accuracy loss.

The recent success of convolutional neural networks (CNN) has led its implementation in specialized accelerators such as graphics processing unit (GPUs). However, the intensive computing workloads of CNNs remain a challenge to existing accelerators. By leveraging the error tolerance of CNNs, we propose a novel method to design energy-efficient CNN accelerators using approximate computation reuse (ACR), referred to as AxRNN. Computation reuse aims to reuse the previously computed results to avoid redundant executions. However, it cannot be applied directly to CNNs because CNNs do not have enough data locality. Thus, AxRNN performs approximate computation reuse under relaxed precision requirements on input patterns and design a reconfigurable architecture to support the ACR. This reconfigurable pattern matching is central to achieve a "controllable approximation". We implement the AxRNN using content addressable memory and integrate them with floating point units. Simulation results show that AxRNN reduces the computation energy by 30-58% with only 1-2.5% accuracy degradation on MNIST, EMNIST, and CIFAR-10 dataset.

As Moore's Law comes to an end and transistor scaling increasingly falls short in improving energy efficiency, alternative computing paradigms are direly needed. This need is further highlighted by the overwhelming increase in computing demand posed by emerging applications, such as multimedia and data analysis. Fortunately, such driving workloads also present new opportunities since, thanks to their inherent error tolerance, they do not require completely accurate computations. Thus, by trading off accuracy for better performance or improved efficiency, approximate computing promises tremendous growth for future computing. Various approximation methods demonstrate the effectiveness of voltage scaling in functional units (FUs) for exploring this energy-error tradeoff. Yet, while an accurate error model is critical for assessing the error behavior of voltage-scaled FUs and its effects on application quality, existing error models of voltage-scaled FUs overlook the effects of input data and error rate disparity among different bits. To tackle this challenge, we propose LEVAX, an input-aware learning-based error model of voltage-scaled FUs that can predict the timing error rate (TER) for each output bit. This model is trained using random forest methods, with input features and output labels extracted from gate-level simulations. To validate its effectiveness and demonstrate its prediction accuracy, we use LEVAX on various FUs. Across all bit positions, voltage levels, and FUs, LEVAX achieves, on average, a relative error of 1.20%. LEVAX also achieves an average per-voltage root mean square error (RMSE) of 1.03% and per-bit RMSE of 1.17%. Exposing this error rate even up to the application level, LEVAX can estimate the quality of four image processing applications under-voltage scaling with an average accuracy of 97.9%. To the best of our knowledge, LEVAX is the first voltage scaling error model of FUs that can incorporate the effects of input data.