Spiking neural networks (SNNs) are increasingly applied to event-based data generated by event cameras due to their asynchronous and sparse properties. Event cameras can inherently respond to the changes in the scene, which is a quite desirable property for action recognition tasks. However, existing works of SNNs for event-based action recognition are still limited. To capture the rich dynamics embedded in event streams, we propose the two-stream SNN that consists of spatial spiking stream and motion spiking stream to address event-based action recognition. To effectively build the two-stream SNN, we present a motion feature aggregation strategy and an attention-based two-stream fusion method. The motion feature aggregation strategy accumulates motion information and groups it into distinct channels for input into the SNN, which can alleviate the dilemma of information loss caused by compact representation. The attention-based two-stream fusion method can fuse the spatial and motion features effectively using the channel-wise attention mechanism, which helps our network to achieve better integration of two-stream information. Extensive experimental results on three event-based action recognition datasets show our proposed two-stream SNN achieves competitive performance with much fewer trainable parameters, which demonstrates the effectiveness of our work in event-based action recognition tasks.

The advantages of high temporal resolution and high dynamic range provided by event cameras are particularly suitable for moving object detection, especially in scenarios with motion blur and extreme lighting conditions. Current popular methods predominantly focus on designing powerful network architectures to extract event features, often neglecting the rationality of event representation design which has been proven to impact significantly on downstream tasks. In particular, current event representations typically rely on fixed hyperparameters, without considering variations in relative motion speed, a key factor in motion-rich scenes captured by event cameras. To tackle this challenge, we propose a gradient-based scaled Timesurface (STS) motivated by the observation of the relationship between motion speeds and the gradient strength, which adaptively rescales the decay factor at different spatial positions. Additionally, we propose a dataset called RotateDigit, which is the first event dataset featuring clear motion level annotations to our best knowledge. Proposed STS method is verified using Spiking Neural Network (SNNs) due to the sharing asynchronous and sparse properties with event camera. Experimental results on RotateDigit and Gen1 show the performance improvement achieved by STS, which validates the rationality and effectiveness of our work.

Spiking neural networks (SNNs) operating with asynchronous discrete events show higher energy efficiency with sparse computation. A popular approach for implementing deep SNNs is artificial neural network (ANN)-SNN conversion combining both efficient training of ANNs and efficient inference of SNNs. However, the accuracy loss is usually nonnegligible, especially under few time steps, which restricts the applications of SNN on latency-sensitive edge devices greatly. In this article, we first identify that such performance degradation stems from the misrepresentation of the negative or overflow residual membrane potential in SNNs. Inspired by this, we decompose the conversion error into three parts: quantization error, clipping error, and residual membrane potential representation error. With such insights, we propose a two-stage conversion algorithm to minimize those errors, respectively. In addition, we show that each stage achieves significant performance gains in a complementary manner. By evaluating on challenging datasets including CIFAR- 10, CIFAR- 100, and ImageNet, the proposed method demonstrates the state-of-the-art performance in terms of accuracy, latency, and energy preservation. Furthermore, our method is evaluated using a more challenging object detection task, revealing notable gains in regression performance under ultralow latency, when compared with existing spike-based detection algorithms. Codes will be available at: https://github.com/Windere/snn-cvt-dual-phase.

Spiking neural networks (SNNs) with bio-inspired spatio-temporal dynamics, have increasingly manifested their superiority in energy efficiency. However, the non-differential spiking activity hinders the implementation of the efficient error backpropagation in training SNNs. The existing solution circumvents this problem by a relaxation on the gradient calculation using a continuous function, which is referred to as surrogate gradient learning. Nevertheless, such a solution leads to a prominent gradient mismatch problem due to the low precision of spikes, which limits the performance of directly trained SNNs on deeper architectures. To tackle this issue, we propose the adaptive multilevel firing (AMLF) method incorporating the spiking dormant-suppressed residual network (spiking DS-ResNet) to get well-behaved SNNs. The AMLF method can not only adaptively implement incremental expression ability of spiking neurons to alleviate gradient mismatch issue, but also enable more efficient gradient propagation during training by enlarging the gradient-available area. We conduct a series of experiments on static image datasets and neuromorphic datasets. Results show that the proposed AMLF method can help SNNs achieve competitive performance in terms of both latency and accuracy.

Spiking neural networks (SNNs) with biologically inspired spatio-temporal dynamics demonstrate superior energy efficiency on neuromorphic architectures. Error backpropagation in SNNs is prohibited by the all-or-none nature of spikes. The existing solution circumvents this problem by a relaxation on the gradient calculation using a continuous function with a constant relaxation de- gree, so-called surrogate gradient learning. Nevertheless, such a solution introduces additional smoothing error on spike firing which leads to the gradients being estimated inaccurately. Thus, how to adaptively adjust the relaxation degree and eliminate smoothing error progressively is crucial. Here, we propose a methodology such that training a prototype neural network will evolve into training an SNN gradually by fusing the learnable relaxation degree into the network with random spike noise. In this way, the network learns adaptively the accurate gradients of loss landscape in SNNs. The theoretical analysis further shows optimization on such a noisy network could be evolved into optimization on the embedded SNN with shared weights progressively. Moreover, The experiments on static images, dynamic event streams, speech, and instrumental sounds show the proposed method achieves state-of-the-art performance across all the datasets with remarkable robustness on different relaxation degrees.

Spiking neural networks (SNNs) have increasingly drawn massive research attention due to biological interpretability and efficient computation. Recent achievements are devoted to utilizing the surrogate gradient (SG) method to avoid the dilemma of non-differentiability of spiking activity to directly train SNNs by backpropagation. However, the fixed width of the SG leads to gradient vanishing and mismatch problems, thus limiting the performance of directly trained SNNs. In this work, we propose a novel perspective to unlock the width limitation of SG, called the learnable surrogate gradient (LSG) method. The LSG method modulates the width of SG according to the change of the distribution of the membrane potentials, which is identified to be related to the decay factors based on our theoretical analysis. Then we introduce the trainable decay factors to implement the LSG method, which can optimize the width of SG automatically during training to avoid the gradient vanishing and mismatch problems caused by the limited width of SG. We evaluate the proposed LSG method on both image and neuromorphic datasets. Experimental results show that the LSG method can effectively alleviate the blocking of gradient propagation caused by the limited width of SG when training deep SNNs directly. Meanwhile, the LSG method can help SNNs achieve competitive performance on both latency and accuracy.

Recent work on spiking neural networks (SNNs) has focused on achieving deep architectures. They commonly use backpropagation (BP) to train SNNs directly, which allows SNNs to go deeper and achieve higher performance. However, the BP training procedure is computing intensive and complicated by many trainable parameters. Inspired by global pooling in convolutional neural networks (CNNs), we present the spike probabilistic global pooling (SPGP) method based on a probability function for training deep convolutional SNNs. It aims to remove the difficulty of too many trainable parameters brought by multiple layers in the training process, which can reduce the risk of overfitting and get better performance for deep SNNs (DSNNs). We use the discrete leaky-integrate-fire model and the spatiotemporal BP algorithm for training DSNNs directly. As a result, our model trained with the SPGP method achieves competitive performance compared to the existing DSNNs on image and neuromorphic data sets while minimizing the number of trainable parameters. In addition, the proposed SPGP method shows its effectiveness in performance improvement, convergence, and generalization ability.