While transformers dominate modern vision and language models, their attention mechanism remains poorly suited for in-memory computing (IMC) devices due to intensive activation-to-activation multiplications and non-local memory access, leading to substantial latency and bandwidth overhead on ReRAM-based accelerators. To address this mismatch, we propose FAR, a Function-preserving Attention Replacement framework that substitutes all attention in pretrained DeiTs with sequential modules inherently compatible with IMC dataflows. Specifically, FAR replaces self-attention with a multi-head bidirectional LSTM architecture via block-wise distillation to retain functional equivalence while enabling linear-time computation and localized weight reuse. We further incorporate structured pruning on FAR models, enabling flexible adaptation to resource-constrained IMC arrays while maintaining functional fidelity. Evaluations on the DeiT family demonstrate that FAR maintains comparable accuracy to the original attention-based models on ImageNet and multiple downstream tasks with reduced parameters and latency. Further analysis shows that FAR preserves the semantic token relationships learned by attention while improving computational efficiency, highlighting its potential for energy-efficient transformer inference on IMC-based edge accelerators.

With storage and computation happening at the same place, computing in resistive crossbars minimizes data movement and avoids the memory bottleneck issue. It leads to ultra-high energy efficiency for data-intensive applications. However, defects in crossbars severely affect computing accuracy. Existing solutions, including re-training with defects and redundant designs, but they have limitations in practical implementations. In this work, we introduce row shuffling and output compensation to mitigate defects without re-training or redundant resistive crossbars. We also analyzed the coupling effects of defects and circuit parasitics. Moreover, We study different combinations of methods to achieve the best trade-off between cost and performance. Our proposed methods could rescue up to 10% defects in ResNet-20 application without performance degradation.

Resistive crossbar arrays are known for their unique structure to implement analog in-memory vector-matrix-multiplications (VMM). However, general-purpose circuit simulators, such as HSPICE and HSIM, are too slow for large scale crossbar array simulations with consideration of circuit parasitics. Although there are some specific simulators designed for crossbar arrays, they mainly focus on area/power/delay estimation rather than accurate SPICE-level simulation, thus could not model its functionality on analog in-memory computing. In this paper, we firstly give a SPICE-level modeling of resistive crossbar array with consideration of circuit parasitics in MATLAB. We also propose efficient methods to further speedup simulations by model simplifications. Last but not least, ResNet-20 on CIFAR-10 is applied to demonstrate the work. With the proposed model simplification methods, simulation speed can be improved by ~31X with tolerable errors, and more than 5X speedup is achieved on ResNet-20 while the accuracy drop is 6%.

Traditional computing hardware often encounters on-chip memory bottleneck on large-scale Convolution Neural Networks (CNN) applications. With its unique in-memory computing feature, resistive crossbar-based computing attracts researchers’ attention as a promising solution to the memory bottleneck issue in von Neumann architectures. However, the parasitic resistances in crossbar deviate its behavior from the ideal weighted summation operation. In large-scale implementations, the impact of parasitic resistances must be carefully considered and mitigated to ensure circuits’ functionality. In this work, we implemented and simulated CNNs on resistive crossbar circuits with consideration of parasitic resistances. Moreover, we carried out a new mapping scheme for high utilization of crossbar arrays on convolution, and a mitigation algorithm to mitigate parasitic resistances in CNN applications. The mitigation algorithm considers parasitic resistances as well as data/kernel patterns of each layer to minimize the computing error in crossbar-based convolutions of CNNs. We demonstrated the proposed methods with implementations of a 4-layer CNN on MNIST, and residual neural network (ResNet) (20, 32, and 56) on CIFAR-10. Simulation results show the proposed methods well mitigate the parasitic resistances in crossbars. With our methods, modern CNNs on crossbars can preserve ideal (software) level classification accuracy with 6-bit ADCs and DACs implementation.

Memristor with tunable non-volatile resistance offers in-memory computing capability that avoids the von-Neumann bottleneck. However, large-scale experimental demonstration to this end is yet to be implemented due to the immaturity of the device and integration technologies. Here in this paper we report our recent process in analog computing using analog-voltage-amplitude-vector input and analog-memristor-conductance matrix, with applications in signal and image processing. The vector matrix multiplication is processed in the memristor crossbars in one step, with 5-8 bit precision depending on the array size. The demonstration is made possible by high memristor yield (99.8%), stable multilevel memresistance states, linear current-voltage (IV) relation in the operation range, and low wire resistance between the cells.

Memristor-based synaptic network has been widely investigated and applied to neuromorphic computing systems for the fast computation and low design cost. As memristors continue to mature and achieve higher density, bit failures within crossbar arrays can become a critical issue. These can degrade the computation accuracy significantly. In this work, we propose a defect rescuing design to restore the computation accuracy. In our proposed design, significant weights in a specified network are first identified and retraining and remapping algorithms are described. For a two layer neural network with 92.64% classification accuracy on MNIST digit recognition, our evaluation based on real device testing shows that our design can recover almost its full performance when 20% random defects are present.

Recent advances in memristor technology lead to the feasibility of large-scale neuromorphic systems by leveraging the similarity between memristor devices and synapses. For instance, memristor cross-point arrays can realize dense synapse network among hundreds of neuron circuits, which is not affordable for traditional implementations. However, little progress was made in synapse designs that support both static and dynamic synaptic properties. In addition, many neuron circuits require signals in specific pulse shape, limiting the scale of system implementation. Last but not least, a bottom-up study starting from realistic memristor devices is still missing in the current research of memristor-based neuromorphic systems. Here, we propose a memristor-based dynamic (MD) synapse design with experiment-calibrated memristor models. The structure obtains both static and dynamic synaptic properties by using one memristor for weight storage and the other as a selector. We overcame the device nonlinearities and demonstrated spike-timing-based recall, weight tunability, and spike-timing-based learning functions on MD synapse. Furthermore, a temporal pattern learning application was investigated to evaluate the use of MD synapses in spiking neural networks, under both spike-timing-dependent plasticity and remote supervised method learning rules.

Vector-matrix multiplication dominates the computation time and energy for many workloads, particularly neural network algorithms and linear transforms (e.g, the Discrete Fourier Transform). Utilizing the natural current accumulation feature of memristor crossbar, we developed the Dot-Product Engine (DPE) as a high density, high power efficiency accelerator for approximate matrix-vector multiplication. We firstly invented a conversion algorithm to map arbitrary matrix values appropriately to memristor conductances in a realistic crossbar array, accounting for device physics and circuit issues to reduce computational errors. The accurate device resistance programming in large arrays is enabled by close-loop pulse tuning and access transistors. To validate our approach, we simulated and benchmarked one of the state-of-the-art neural networks for pattern recognition on the DPEs. The result shows no accuracy degradation compared to software approach (99 % pattern recognition accuracy for MNIST data set) with only 4 Bit DAC/ADC requirement, while the DPE can achieve a speed-efficiency product of 1,000× to 10,000× compared to a custom digital ASIC.

A number of recent efforts have attempted to design accelerators for popular machine learning algorithms, such as those involving convolutional and deep neural networks (CNNs and DNNs). These algorithms typically involve a large number of multiply-accumulate (dot-product) operations. A recent project, DaDianNao, adopts a near data processing approach, where a specialized neural functional unit performs all the digital arithmetic operations and receives input weights from adjacent eDRAM banks. This work explores an in-situ processing approach, where memristor crossbar arrays not only store input weights, but are also used to perform dot-product operations in an analog manner. While the use of crossbar memory as an analog dot-product engine is well known, no prior work has designed or characterized a full-fledged accelerator based on crossbars. In particular, our work makes the following contributions: (i) We design a pipelined architecture, with some crossbars dedicated for each neural network layer, and eDRAM buffers that aggregate data between pipeline stages. (ii) We define new data encoding techniques that are amenable to analog computations and that can reduce the high overheads of analog-to-digital conversion (ADC). (iii) We define the many supporting digital components required in an analog CNN accelerator and carry out a design space exploration to identify the best balance of memristor storage/compute, ADCs, and eDRAM storage on a chip. On a suite of CNN and DNN workloads, the proposed ISAAC architecture yields improvements of 14.8×, 5.5×, and 7.5× in throughput, energy, and computational density (respectively), relative to the state-of-the-art DaDianNao architecture.

Memristor, the fourth basic circuit element, demonstrates obvious stochastic behaviors in both the static resistance states and the dynamic switching. In this work, a novel memristor-based true random number generator (MTRNG) is presented which leverages the stochastic property when switching a device between its binary states. Compared to conventional random number generators that require amplifiers or comparators with high complexity, the use of memristors significantly reduces the design cost: a basic MTRNG consists of only one memristor, six transistors, and one D Flip-flop. To maximize the entropy of the random bit generation, we further enhanced the design to a 2-branch scheme which can provide a uniform bit distribution. Our simulation results show that the proposed MTRNGs offer high operating speed and low power consumption: the reading clocks of the basic 1-branch and the enhanced 2-branch schemes can reach at 1.05GHz and 0.96GHz with power assumptions of 31.1"W and 80.3"W, respectively. Moreover, the zero-versus-one distributions and sampling rates of MTRNGs can be flexibly reconfigured by modulating the width and amplitude of the programming pulse applied on a memristor and therefore adjusting its switching probability between ON and OFF states.

Memristor, the fourth basic circuit element, has shown great potential in neuromorphic circuit design for its unique synapse-like feature. However, though the continuous resistance state of memristor has been expected, obtaining and maintaining an arbitrary intermediate state cannot be well controlled in nowadays memristive system. In addition, the stochastic switching behaviors have been widely observed. To facilitate the investigation on memristor-based hardware implementation, we built a stochastic behavior model of TiO2memristive devices based on the real experimental results. By leveraging the stochastic behavior of memristors, a macro cell design composed of multiple parallel connecting memristors can be successfully used in implementing the weight storage unit and the stochastic neuron — the two fundamental components in neural network (NN)s, providing a feasible solution in memristor-based hardware implementation.

By mimicking the highly parallel biological systems, neuromorphic hardware provides the capability of information processing within a compact and energy-efficient platform. However, traditional Von Neumann architecture and the limited signal connections have severely constrained the scalability and performance of such hardware implementations. Recently, many research efforts have been investigated in utilizing the latest discovered memristors in neuromorphic systems due to the similarity of memristors to biological synapses. In this paper, we explore the potential of a memristor crossbar array that functions as an autoassociative memory and apply it to brain-state-in-a-box (BSB) neural networks. Especially, the recall and training functions of a multianswer character recognition process based on the BSB model are studied. The robustness of the BSB circuit is analyzed and evaluated based on extensive Monte Carlo simulations, considering input defects, process variations, and electrical fluctuations. The results show that the hardware-based training scheme proposed in the paper can alleviate and even cancel out the majority of the noise issue.

In this work, we propose a hardware realization of the Brain-State-in-a-Box (BSB) neural network model training algorithm. This method can be implemented as an analog/digital mixed-signal circuit to train memristor crossbar arrays within BSB circuits. The training effect is demonstrated through experimentation and the quality as an auto-associative memory is also analyzed and compared with software based training methods. The impacts of non-ideal device characteristics and fabrication defects in crossbar arrays are discussed. Our hardware architecture shows great potential for low power, high speed, small hardware size computations, and provides inherent security features.

The invention of neuromorphic computing architecture is inspired by the working mechanism of human-brain. Memristor technology revitalized neuromorphic computing system design by efficiently executing the analog Matrix-Vector multiplication on the memristor-based crossbar (MBC) structure. However, programming the MBC to the target state can be very challenging due to the difficulty to real-time monitor the memristor state during the training. In this work, we quantitatively analyzed the sensitivity of the MBC programming to the process variations and input signal noise. We then proposed a noise-eliminating training method on top of a new crossbar structure to minimize the noise accumulation during the MBC training and improve the trained system performance, i.e., the pattern recall rate. A digital-assisted initialization step for MBC training is also introduced to reduce the training failure rate as well as the training time. Experimental results show that our noise-eliminating training method can improve the pattern recall rate. For the tested patterns with 128 x 128 pixels our technique can reduce the MBC training time by 12.6% ~ 14.1% for the same pattern recognition rate, or improve the pattern recall rate by 18.7% ~ 36.2% for the same training time.

The cessation of Moore's Law has limited further improvements in power efficiency. In recent years, the physical realization of the memristor has demonstrated a promising solution to ultra-integrated hardware realization of neural networks, which can be leveraged for better performance and power efficiency gains. In this work, we introduce a power efficient framework for approximated computations by taking advantage of the memristor-based multilayer neural networks. A programmable memristor approximated computation unit (Memristor ACU) is introduced first to accelerate approximated computation and a memristor-based approximated computation framework with scalability is proposed on top of the Memristor ACU. We also introduce a parameter configuration algorithm of the Memristor ACU and a feedback state tuning circuit to program the Memristor ACU effectively. Our simulation results show that the maximum error of the Memristor ACU for 6 common complex functions is only 1.87% while the state tuning circuit can achieve 12-bit precision. The implementation of HMAX model atop our proposed memristor-based approximated computation framework demonstrates 22× power efficiency improvements than its pure digital implementation counterpart.

The Brain-State-in-a-Box (BSB) model is an auto-associative neural network that has been widely used in optical character recognition and image processing. Traditionally, the BSB model was realized at software level and carried out on high-performance computing clusters. To improve computation efficiency and reduce resources requirement, we propose a hardware realization by utilizing memristor crossbar arrays. In this work, we explore the potential of a memristor crossbar array as an auto-associative memory. More specificly, the recall function of a multi-answer character recognition based on BSB model was realized. The robustness of the proposed BSB circuit was analyzed and evaluated based on massive Monte-Carlo simulations, considering input defects, process variations, and electrical fluctuations. The physical constrains when implementing a neural network with memristor crossbar array have also been discussed. Our results show that the BSB circuit has a high tolerance to random noise. Comparably, the correlations between memristor arrays introduces directional noise and hence dominates the quality of circuits.

Memristor, the fourth passive circuit element, has attracted increased attention since it was rediscovered by HP Lab in 2008. Its distinctive characteristic to record the historic profile of the voltage/current creates a great potential for future neuromorphic computing system design. However, at the nano-scale, process variation control in the manufacturing of memristor devices is very difficult. The impact of process variations on a memristive system that relies on the continuous (analog) states of the memristors could be significant. We use TiO2-based memristor as an example to analyze the impact of geometry variations on the electrical properties. A simple algorithm was proposed to generate a large volume of geometry variation-aware three-dimensional device structures for Monte-Carlo simulations. A neuromorphic computing system based on memristor-based bidirectional synapse design is proposed as case study. We analyze and evaluate the robustness of the proposed system in pattern recognition based on massive Monte-Carlo simulations, after considering input defects and process variations.

The Brain-State-in-a-Box (BSB) model is an auto-associative neural network that has been widely used in optical character recognition and image processing. Traditionally, the BSB model was realized at software level and carried out on high-performance computing clusters. To improve computation efficiency and reduce resource requirement, we propose a hardware realization by utilizing memristor crossbar arrays. Memristors can remember the historical profiles of the excitations and record them as analog variables. The similarity to biological synaptic behavior has encouraged a lot of research on memristor-based neuromorphic hardware system. In this work, we explore the potential of a memristor crossbar array as an auto-associative memory. More specifically, the recall function of a multi-answer character recognition based on BSB model was realized. The robustness of the proposed BSB circuit was analyzed and evaluated based on massive Monte-Carlo simulations, considering input defects, process variations, and electrical fluctuations. The physical constraints when implementing a neural network with memristor crossbar array have also been discussed. Our results show that the BSB circuit has a high tolerance to random noise. Comparably, the correlations between memristor arrays introduce directional noise and hence dominate the quality of the circuit.

The fourth passive circuit element, memristor, has attracted increased attentions since the first real device was discovered by HP Lab in 2008. Its distinctive characteristic to record the historic profile of the voltage/current through itself creates great potentials in future system design. However, as a nano-scale device, memristor is facing great challenge on process variation control in the manufacturing. In this work, we analyze the impact of the geometry variations on the electrical properties of both TiO2thin-film and spintronic memristors, including line edge roughness and thickness fluctuation. A simple algorithm was proposed to generate a large volume of geometry variation-aware three-dimensional device structures for Monte-Carlo simulations. Our simulation results show that due to the different physical mechanisms, TiO2thin-film memristor and spintronic memristor demonstrate very different electrical characteristics even when exposing them to the same excitations and under the same process variation conditions.

The emerging memristor devices have recently received increased attention since HP Lab reported the first TiO2-based memristive structure. As it is at nano-scale geometry size, the uniformity of memristor device is difficult to control due to the process variations in the fabrication process. The incurred design concerns in a memristor-based computing system, e.g, neuromorphic computing, can be very severe because the analog states of memristors are heavily utilized. Therefore, the understanding and quantitative characterization of the impact of process variations on the electrical properties of memristors become crucial for the corresponding VLSI designs. In this work, we examined the theoretical model of TiO2thin-film memristors and studied the relationships between the electrical parameters and the process variations of the devices. A statistical model based on a process-variation aware memristor device structure is extracted accordingly. Simulations show that our proposed model is 3 ~ 4 magnitude faster than the existing Monte-Carlo simulation method, with only ~ 2% accuracy degradation. A variable gain amplifier (VGA) is used as the case study to demonstrate the applications of our model in memristor-based circuit designs.

In this paper, we present a compact model of the spintronic memristor based on the magnetic-domain-wall motion mechanism for circuit design. Our model also takes into account the variations of material parameters and fabrication process, which significantly affects the actual electrical characteristics of a memristor in nano-scale technologies. Our proposed model can be easily implemented by Verilog-A languages and compatible to SPICE-based simulation. Based on our model, we also show some potential applications of memristor in computing system, including the detailed analysis and optimizations based on our proposed model.