Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch. This paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA). Essentially, GSA comprises a two-layer GLA linked via $\operatorname{softmax}$, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size. This design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size. Additionally, retaining the $\operatorname{softmax}$ operation is particularly beneficial in ``finetuning pretrained Transformers to RNNs'' (T2R) settings, reducing the need for extensive training from scratch. Extensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings.

We present a fully memristive spiking neural network (MSNN) consisting of novel memristive neurons trained using the backpropagation through time (BPTT) learning rule. Gradient descent is applied directly to the memristive integrate-and-fire (MIF) neuron designed using analog SPICE circuit models, which generates distinct depolarization, hyperpolarization, and repolarization voltage waveforms. Synaptic weights are trained by BPTT using the membrane potential of the MIF neuron model and can be processed on memristive crossbars. The natural spiking dynamics of the MIF neuron model are fully differentiable, eliminating the need for gradient approximations that are prevalent in the spiking neural network literature. Despite the added complexity of training directly on SPICE circuit models, we achieve 97.58% accuracy on the MNIST test set and 75.26% on the Fashion-MNIST test set, which is considerably high among all fully MSNNs with small-scale neural networks.

We present a fully memristive spiking neural network (MSNN) consisting of physically-realizable memristive neurons and memristive synapses to implement an unsupervised Spike Timing Dependent Plasticity (STDP) learning rule. The system is fully memristive in that both neuronal and synaptic dynamics can be realized by using memristors. The neuron is implemented using the SPICE-level memristive integrate-and-fire (MIF) model, which consists of a minimal number of circuit elements necessary to achieve distinct depolarization, hyperpolarization, and repolarization voltage waveforms. The proposed MSNN uniquely implements STDP learning by using cumulative weight changes in memristive synapses from the voltage waveform changes across the synapses, which arise from the presynaptic and postsynaptic spiking voltage signals during the training process. Two types of MSNN architectures are investigated: 1) a biologically plausible memory retrieval system, and 2) a multi-class classification system. Our circuit simulation results verify the MSNN’s unsupervised learning efficacy by replicating biological memory retrieval mechanisms, and achieving 97.5% accuracy in a 4-pattern recognition problem in a large scale discriminative MSNN.

We present and experimentally validate two minimal compact memristive models for spiking neuronal signal generation using commercially available low-cost components. The first neuron model is called the Memristive Integrate-and-Fire (MIF) model, for neuronal signaling with two voltage levels: the spike-peak, and the rest-potential. The second model MIF2 is also presented, which promotes local adaptation by accounting for a third refractory voltage level during hyperpolarization. We show both compact models are minimal in terms of the number of circuit elements and integration area. Using the MIF and MIF2 models, we postulate the design of a memristive solid-state brain with an estimation of its surface area and power consumption. Analytical projections show that a memristive solid-state brain could be realized within (i) the surface area of the median human brain, 2,400cm2, (ii) the same volume of the median human brain, and (iii) a total power budget of approximately 20 W using a 3.5 nm technology. Distinct from the past decade of memristive neuron literature, our benchmarks are attained using generic commercially available memristors that are reproducible using off-the-shelf components. We expect this work can promote more experimental demonstrations of memristive circuits that do not rely on prohibitively expensive fabrication processes.