On-device learning has emerged as a prevailing trend that avoids the slow response time and costly communication of cloud-based learning. The ability to learn continuously and indefinitely in a changing environment, and with resource constraints, is critical for real sensor deployments. However, existing designs are inadequate for practical scenarios with (i) streaming data input, (ii) lack of supervision and (iii) limited on-board resources. In this paper, we design and deploy the first on-device lifelong learning system called LifeHD for general IoT applications with limited supervision. LifeHD is designed based on a novel neurally-inspired and lightweight learning paradigm called Hyperdimensional Computing (HDC). We utilize a two-tier associative memory organization to intelligently store and manage high-dimensional, low-precision vectors, which represent the historical patterns as cluster centroids. We additionally propose two variants of LifeHD to cope with scarce labeled inputs and power constraints. We implement LifeHD on off-the-shelf edge platforms and perform extensive evaluations across three scenarios. Our measurements show that LifeHD improves the unsupervised clustering accuracy by up to 74.8% compared to the state-of-the-art NN-based unsupervised lifelong learning baselines with as much as 34.3x better energy efficiency. Our code is available at https://github.com/Orienfish/LifeHD.

We propose a normative model for spatial representation in the hippocampal formation that combines optimality principles, such as maximizing coding range and spatial information per neuron, with an algebraic framework for computing in distributed representation. Spatial position is encoded in a residue number system, with individual residues represented by high-dimensional, complex-valued vectors. These are composed into a single vector representing position by a similarity-preserving, conjunctive vector-binding operation. Self-consistency between the vectors representing position and the individual residues is enforced by a modular attractor network whose modules correspond to the grid cell modules in entorhinal cortex. The vector binding operation can also be used to bind different contexts to spatial representations, yielding a model for entorhinal cortex and hippocampus. We provide model analysis of scaling, similarity preservation and convergence behavior as well as experiments demonstrating noise robustness, sub-integer resolution in representing position, and path integration. The model formalizes the computations in the cognitive map and makes testable experimental predictions.

Today’s machine learning (ML) systems, running workloads, such as deep neural networks, which require billions of parameters and many hours to train a model, consume a significant amount of energy. Due to the complexity of computation and topology, even the quantized models are hard to deploy on edge devices under energy constraints. To combat this, researchers have been focusing on new emerging neuromorphic computing models. Two of those models are hyperdimensional computing (HDC) and spiking neural networks (SNNs), both with their own benefits. HDC has various desirable properties that other ML algorithms lack, such as robustness to noise, simple operations, and high parallelism. SNNs are able to process event-based signal data in an efficient manner. This work develops$\mathsf {HyperSpike}$, which utilizes a single, randomly initialized, and untrained SNN layer as a feature extractor connected to a trained HDC classifier. HDC is used to enable more efficient classification as well as provide robustness to errors. We experimentally show that$\mathsf {HyperSpike}$is on average$31.5\times $more robust to errors than traditional SNNs. On Intel’s Loihi (Davies et al., 2018),$\mathsf {HyperSpike}$is$10\times $faster and$2.6\times $more energy efficient over traditional SNN networks. We further develop$\mathsf {HyperSpikeASIC}$, a customized accelerator for$\mathsf {HyperSpike}$. By decoupling the neuron and synapses,$\mathsf {HyperSpikeASIC}$skips the inactive neurons and limits the neuron state updating to once per time step at most.$\mathsf {HyperSpikeASIC}$is$601\times $faster and$3467\times $more energy efficient than$\mathsf {HyperSpike}$running on Intel’s Loihi for SNN acceleration, and$12.2\times $faster and$211\times $more energy efficient than the state-of-the-art SNN ASIC implementation (Wang et al., 2022).

Today's Machine Learning(ML) systems, especially those running in server farms running workloads such as Deep Neural Networks, which require billions of parameters and many hours to train a model, consume a significant amount of energy. To combat this, researchers have been focusing on new emerging neuromorphic computing models. Two of those models are Hyperdimensional Computing (HDC) and Spiking Neural Networks (SNNs), both with their own benefits. HDC has various desirable properties that other Machine Learning (ML) algorithms lack such as: robustness to noise in the system, simple operations, and high parallelism. SNNs are able to process event based signal data in an efficient manner. In this paper, we create HyperSpike, which utilizes a single, randomly initialized and untrained SNN layer as feature extractor connected to a trained HDC classifier. HDC is used to enable more efficient classification as well as provide robustness to errors. We experimentally show that HyperSpike is on average 31.5× more robust to errors than traditional SNNs. We also implement HyperSpike in hardware, and show that it is 10x faster and 2.6× more energy efficient over traditional SNN networks run on Intel's Loihi [1].

Advances in graph neural network (GNN)-based algorithms enable machine learning on relational data. GNNs are computationally demanding since they rely upon backpropagation over the graph data that has sparse and irregular characteristics. In this paper, we propose a lightweight graph-based machine learning framework based on hyperdimensional computing (HDC) called RelHD. It maps the features of each node into a high-dimensional space and embeds relationships between nodes. Using lightweight HDC operations, RelHD enables both training and inference on graph data without backpropagation. Furthermore, we design a scalable processing in-memory (PIM) architecture based on the emerging FeFET technology to accelerate the proposed algorithm. Our strategy optimizes data allocation and operation scheduling that maximizes the accelerator performance by addressing the sparseness and irregularity of the graph. Experimental results show that RelHD offers comparable accuracy to the popular GNN-based algorithms while being up to 32× faster on GPU. Also, our FeFET-based accelerator achieves 33× of speedup and 59287× energy efficiency improvement on average over the GPU. It is 10× faster and 986× more energy efficient on average compared to the state-of-the-art in-memory processing-based GNN accelerator.

Hyperdimensional (HD) computing is a set of neurally inspired methods for computing on high-dimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. We present a novel mathematical framework that unifies analysis of HD computing architectures, and provides general, non-asymptotic, sufficient conditions under which HD information processing techniques will succeed.

Hyperdimensional (HD) computing is a set of neurally inspired methods for obtaining highdimensional, low-precision, distributed representations of data. These representations can be combined with simple, neurally plausible algorithms to effect a variety of information processing tasks. HD computing has recently garnered significant interest from the computer hardware community as an energy-efficient, low-latency, and noise-robust tool for solving learning problems. In this review, we present a unified treatment of the theoretical foundations of HD computing with a focus on the suitability of representations for learning.

Hyperdimensional computing (HD) is an emerging paradigm for machine learning based on the evidence that the brain computes on high-dimensional, distributed, representations of data. The main operation of HD is encoding, which transfers the input data to hyperspace by mapping each input feature to a hypervector, followed by a bundling procedure that adds up the hypervectors to realize the encoding hypervector. The operations of HD are simple and highly parallelizable, but the large number of operations hampers the efficiency of HD in embedded domain. In this paper, we propose SHEARer, an algorithmhardware co-optimization to improve the performance and energy consumption of HD computing. We gain insight from a prudent scheme of approximating the hypervectors that, thanks to error resiliency of HD, has minimal impact on accuracy while provides high prospect for hardware optimization. Unlike previous works that generate the encoding hypervectors in full precision and then and then perform ex-post quantization, we compute the encoding hypervectors in an approximate manner that saves resources yet affords high accuracy. We also propose a novel FPGA architecture that achieves striking performance through massive parallelism with low power consumption. Moreover, we develop a software framework that enables training HD models by emulating the proposed approximate encodings. The FPGA implementation of SHEARer achieves an average throughput boost of 104,904× (15.7×) and energy savings of up to 56,044× (301×) compared to state-of-the-art encoding methods implemented on Raspberry Pi 3 (GeForce GTX 1080 Ti) using practical machine learning datasets.

Hyperdimensional (HD) computing is a mathematical framework, inspired by neuroscience, which can be used to represent many machine learning (ML) problems. Data is first encoded into high dimensional space (on the order of 103or 104dimensions) to create hypervectors. HD computing combines these hypervectors to create a model used for inference. However, due to the high dimensionality of the hypervectors, inference in HD is very expensive, especially when it runs on embedded devices with limited resources. One naive approach to improve the efficiency of HD computing is to simply lower the dimensionality of hypervectors, which comes with a corresponding loss in accuracy. However, if the data is compressed intelligently, we can reduce the dimensionality of an HD model without sacrificing accuracy. To that end, we propose CompHD, a novel approach for compressing HD models while maintaining the accuracy of the original model. CompHD utilizes the mathematics of high-dimensional spaces to compress hypervectors into shorter vectors while maintaining the information of full length hypervectors. We evaluated the efficiency of CompHD on a variety of applications. Our results show that CompHD can reduce model size by an average of 69.7%, resulting in a execution time speed up of 4.1 × and improving energy efficiency by 74% while maintaining the accuracy of the original model. This enables more low powered IoT devices to utilize HD computing for ML problems.

The growing use of statistical and machine learning (ML) algorithms to analyze large datasets has given rise to new systems to scale such algorithms. But implementing new scalable algorithms in low-level languages is a painful process, especially for enterprise and scientific users. To mitigate this issue, a new breed of systems expose high-level bulk linear algebra (LA) primitives that are scalable. By composing such LA primitives, users can write analysis algorithms in a higher-level language, while the system handles scalability issues. But there is little work on a unified comparative evaluation of the scalability, efficiency, and effectiveness of such "scalable LA systems." We take a major step towards filling this gap. We introduce a suite of LA-specific tests based on our analysis of the data access and communication patterns of LA workloads and their use cases. Using our tests, we perform a comprehensive empirical comparison of a few popular scalable LA systems: MADlib, MLlib, SystemML, ScaLAPACK, SciDB, and TensorFlow using both synthetic data and a large real-world dataset. Our study has revealed several scalability bottlenecks, unusual performance trends, and even bugs in some systems. Our findings have already led to improvements in SystemML, with other systems' developers also expressing interest. All of our code and data scripts are available for download at https://adalabucsd.github.io/slab.html.