Domain-specific accelerators deliver exceptional performance on their target workloads through fabrication-time orchestrated datapaths. However, such specialized architectures often exhibit performance fragility when exposed to new kernels or irregular input patterns. In contrast, programmable architectures like FPGAs, CGRAs, and GPUs rely on compile-time orchestration to support a broader range of applications; but they are typically less efficient under irregular or sparse data. Pushing the boundaries of programmable architectures requires designs that can achieve efficiency and high-performance on par with specialized accelerators while retaining the agility of general-purpose architectures.

For reasons such as privacy, there are use cases for language models at the edge. This has given rise to small language models targeted for deployment in resource-constrained devices where energy efficiency is critical. Spiking neural networks (SNNs) offer a promising solution due to their energy efficiency, and there are already works on realizing transformer-based models on SNNs. However, key operations like softmax and layer normalization (LN) are difficult to implement on neuromorphic hardware, and many of these early works sidestepped them. To address these challenges, we introduce Sorbet, a transformer-based spiking language model that is more neuromorphic hardware-compatible. Sorbet incorporates a novel shifting-based softmax called PTsoftmax and a BitShifting-based PowerNorm (BSPN), both designed to replace the respective energy-intensive operations. By leveraging knowledge distillation and model quantization, Sorbet achieved a highly compressed binary weight model that maintains competitive performance while achieving $27.16\times$ energy savings compared to BERT. We validate Sorbet through extensive testing on the GLUE benchmark and a series of ablation studies, demonstrating its potential as an energy-efficient solution for language model inference. Our code is publicly available at https://github.com/Kaiwen-Tang/Sorbet

Spiking neural networks (SNNs) are well suited for resource-constrained applications as they do not need expensive multipliers. In a typical rate-encoded SNN, a series of binary spikes within a globally fixed time window is used to fire the neurons. The time window size is also the latency of the network in performing a single inference, as well as determining the overall energy efficiency of the model. The aim of this article is to reduce this while maintaining accuracy when converting artificial neural networks (ANNs) to their equivalent SNNs. The state-of-the-art conversion schemes yield SNNs with accuracies comparable with ANNs only for large window sizes. In this article, we start with understanding the information loss when converting from preexisting ANN models to standard rate-encoded SNN models. From these insights, we propose a suite of techniques that includes a novel SNN encoding scheme, a new spike generation model, an input channel expansion strategy, and a threshold training technique. Together, these methods enabled us to achieve state-of-the-art accuracies using the lowest latencies reported in the literature. In particular, our method achieved a top-1 SNN accuracy of 98.73% (using a single time step) on the MNIST dataset, 76.38% (with eight time steps) on the CIFAR-100 dataset, and 93.71% (eight time steps) on the CIFAR-10 dataset. On ImageNet, an SNN accuracy of 81.9% was achieved using 40 time steps.

With the development of deep learning models, there has been growing research interest in spiking neural networks (SNNs) due to their energy efficiency resulting from their multiplier-less nature. The existing methodologies for SNN development include the conversion of artificial neural networks (ANNs) into equivalent SNNs or the emulation of ANNs, with two crucial challenges yet remaining. The first challenge involves preserving the accuracy of the original ANN models during the conversion to SNNs. The second challenge is to run complex SNNs with lower latencies. To solve the problem of high latency while maintaining high accuracy, we proposed a parallel spike-generation (PSG) method to generate all the spikes in a single timestep, while achieving a better model performance than the standard Integrate-and-Fire model. Based on PSG, we propose OneSpike, a highly effective framework that helps to convert any rate-encoded convolutional SNN into one that uses only one timestep without accuracy loss. Our OneSpike model achieves a state-of-the-art (for SNN) accuracy of 81.92% on the ImageNet dataset using just a single time step. To the best of our knowledge, this study is the first to explore converting multi-timestep SNNs into equivalent single-timestep ones, while maintaining accuracy. These results highlight the potential of our approach in addressing the key challenges in SNN research, paving the way for more efficient and accurate SNNs in practical applications.1

Spiking neural networks (SNNs) are attractive for energy-constrained use-cases due to their binarized activation, eliminating the need for weight multiplication. However, its lag in accuracy compared to traditional convolutional network networks (CNNs) has limited its deployment. In this paper, we propose CQ+ training (extended "clamped" and "quantized" training), an SNN-compatible CNN training algorithm that achieves state-of-the-art accuracy for both CIFAR-10 and CIFAR-100 datasets. Using a 7-layer modified VGG model (VGG-*), we achieved 95.06% accuracy on the CIFAR-10 dataset for equivalent SNNs. The accuracy drop from converting the CNN solution to an SNN is only 0.09% when using a time step of 600. To reduce the latency, we propose a parameterized input encoding method and a threshold training method, which further reduces the time window size to 64 while still achieving an accuracy of 94.09%. For the CIFAR-100 dataset, we achieved an accuracy of 77.27% using the same VGG-* structure and a time window of 500. We also demonstrate the transformation of popular CNNs, including ResNet (basic, bottleneck, and shortcut block), MobileNet v1/2, and Densenet, to SNNs with near-zero conversion accuracy loss and a time window size smaller than 60. The framework was developed in PyTorch and is publicly available.

Spiking neural networks (SNNs) significantly reduce energy consumption by replacing weight multiplications with additions. This makes SNNs suitable for energy-constrained platforms. However, due to its discrete activation, training of SNNs remains a challenge. A popular approach is to first train an equivalent CNN using traditional backpropagation, and then transfer the weights to the intended SNN. Unfortunately, this often results in significant accuracy loss, especially in deeper networks. In this paper, we propose CQ training (Clamped and Quantized training), an SNN-compatible CNN training algorithm with clamp and quantization that achieves near-zero conversion accuracy loss. Essentially, CNN training in CQ training accounts for certain SNN characteristics. Using a 7 layer VGG-* and a 21 layer VGG-19, running on the CIFAR-10 dataset, we achieved 94.16% and 93.44% accuracy in the respective equivalent SNNs. It outperforms other existing comparable works that we know of. We also demonstrate the low-precision weight compatibility for the VGG-19 structure. Without retraining, an accuracy of 93.43% and 92.82% using quantized 9-bit and 8-bit weights, respectively, was achieved. The framework was developed in PyTorch and is publicly available.