The surrogate gradient (SG) method has shown significant promise in enhancing the performance of deep spiking neural networks (SNNs), but it also introduces vulnerabilities to adversarial attacks. Although spike coding strategies and neural dynamics parameters have been extensively studied for their impact on robustness, the critical role of gradient magnitude, which reflects the model's sensitivity to input perturbations, remains underexplored. In SNNs, the gradient magnitude is primarily determined by the interaction between the membrane potential distribution (MPD) and the SG function. In this study, we investigate the relationship between the MPD and SG and their implications for improving the robustness of SNNs. Our theoretical analysis reveals that reducing the proportion of membrane potentials lying within the gradient-available range of the SG function effectively mitigates the sensitivity of SNNs to input perturbations. Building upon this insight, we propose a novel MPD-driven surrogate gradient regularization (MPD-SGR) method, which enhances robustness by explicitly regularizing the MPD based on its interaction with the SG function. Extensive experiments across multiple image classification benchmarks and diverse network architectures confirm that the MPD-SGR method significantly enhances the resilience of SNNs to adversarial perturbations and exhibits strong generalizability across diverse network configurations, SG functions, and spike encoding schemes.

Brain-inspired spiking neural networks (SNNs) are recognized as a promising avenue for achieving efficient, low-energy neuromorphic computing. Direct training of SNNs typically relies on surrogate gradient (SG) learning to estimate derivatives of non-differentiable spiking activity. However, during training, the distribution of neuronal membrane potentials varies across timesteps and progressively deviates toward both sides of the firing threshold. When the firing threshold and SG remain fixed, this may lead to imbalanced spike firing and diminished gradient signals, preventing SNNs from performing well. To address these issues, we propose a novel dual-stage synergistic learning algorithm that achieves forward adaptive thresholding and backward dynamic SG. In forward propagation, we adaptively adjust thresholds based on the distribution of membrane potential dynamics (MPD) at each timestep, which enriches neuronal diversity and effectively balances firing rates across timesteps and layers. In backward propagation, drawing from the underlying association between MPD, threshold, and SG, we dynamically optimize SG to enhance gradient estimation through spatio-temporal alignment, effectively mitigating gradient information loss. Experimental results demonstrate that our method achieves significant performance improvements. Moreover, it allows neurons to fire stable proportions of spikes at each timestep and increases the proportion of neurons that obtain gradients in deeper layers.

Event-based object detection aims to directly recognize and localize objects from asynchronous event streams, facilitating reliable perception in scenarios with fast motion and complex lighting conditions. Its performance largely depends on how temporal information is organized during event sampling. In such dynamic scenarios, fixed sampling strategies struggle to adapt to temporal variations in event streams, whereas recent learnable approaches often exhibit limitations in either their coupling with downstream detection objectives or their ability to fully exploit temporal cues. In this paper, we propose MAS-SNN, a memory-augmented Spiking Neural Network (SNN) framework for event-based object detection, which formulates event sampling as a task-driven, learnable process tightly integrated with representation learning. MAS-SNN introduces a Spiking Memory Attention Embedding (SMAE), composed of a Spiking Memory Buffer (SMB) and a Spiking Memory Attention (SMA) mechanism, to preserve and selectively reuse historical spiking states, thereby fully leveraging the intrinsic temporal dynamics of spiking neurons while maintaining the inherent sparsity and asynchronicity of event-driven vision. MAS-SNN achieves superior performance on N-Caltech 101 and Gen1, demonstrating the effectiveness of memory augmented spiking sampling for event-based object detection.

Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational resource requirements for agents and improve the algorithm's performance under resource-constrained conditions. However, in current spiking reinforcement learning (SRL) algorithms, the simulation results of multiple time steps can only correspond to a single-step decision in RL. This is quite different from the real temporal dynamics in the brain and also fails to fully exploit the capacity of SNNs to process temporal data. In order to address this temporal mismatch issue and further take advantage of the inherent temporal dynamics of spiking neurons, we propose a novel temporal alignment paradigm (TAP) that leverages the single-step update of spiking neurons to accumulate historical state information in RL and introduces gated units to enhance the memory capacity of spiking neurons. Experimental results show that our method can solve partially observable Markov decision processes (POMDPs) and multi-agent cooperation problems with similar performance as recurrent neural networks (RNNs) but with about 50\% power consumption.

Spiking neural networks (SNNs) are increasingly applied to event-based data generated by event cameras due to their asynchronous and sparse properties. Event cameras can inherently respond to the changes in the scene, which is a quite desirable property for action recognition tasks. However, existing works of SNNs for event-based action recognition are still limited. To capture the rich dynamics embedded in event streams, we propose the two-stream SNN that consists of spatial spiking stream and motion spiking stream to address event-based action recognition. To effectively build the two-stream SNN, we present a motion feature aggregation strategy and an attention-based two-stream fusion method. The motion feature aggregation strategy accumulates motion information and groups it into distinct channels for input into the SNN, which can alleviate the dilemma of information loss caused by compact representation. The attention-based two-stream fusion method can fuse the spatial and motion features effectively using the channel-wise attention mechanism, which helps our network to achieve better integration of two-stream information. Extensive experimental results on three event-based action recognition datasets show our proposed two-stream SNN achieves competitive performance with much fewer trainable parameters, which demonstrates the effectiveness of our work in event-based action recognition tasks.

The advantages of high temporal resolution and high dynamic range provided by event cameras are particularly suitable for moving object detection, especially in scenarios with motion blur and extreme lighting conditions. Current popular methods predominantly focus on designing powerful network architectures to extract event features, often neglecting the rationality of event representation design which has been proven to impact significantly on downstream tasks. In particular, current event representations typically rely on fixed hyperparameters, without considering variations in relative motion speed, a key factor in motion-rich scenes captured by event cameras. To tackle this challenge, we propose a gradient-based scaled Timesurface (STS) motivated by the observation of the relationship between motion speeds and the gradient strength, which adaptively rescales the decay factor at different spatial positions. Additionally, we propose a dataset called RotateDigit, which is the first event dataset featuring clear motion level annotations to our best knowledge. Proposed STS method is verified using Spiking Neural Network (SNNs) due to the sharing asynchronous and sparse properties with event camera. Experimental results on RotateDigit and Gen1 show the performance improvement achieved by STS, which validates the rationality and effectiveness of our work.

Reinforcement Learning (RL) has shown promise in robotic navigation tasks, yet applying it to real-world environments remains challenging due to dynamic complexities and the need for dynamically feasible actions. We propose a hierarchical control framework based on Spiking Deep Reinforcement Learning (SDRL) for robust robot navigation in real environments. Our approach utilizes a two-layer architecture: a high-level decision layer powered by a Spiking GRU network for handling partially observable environments, and a low-level executive layer employing Continuous Attractor Neural Networks (CANNs) to ensure precise and continuous actions. This hierarchical structure allows real-time decisionmaking that respects the physical constraints of the robot. Experimental results show that our method adapts effectively to new environments without fine-tuning and surpasses existing methods in performance. We also explore the implementation on the Darwin3 chip, paving the way for biologically inspired motion control in future robotic applications.

Recent advancements have focused on directly training high-performance spiking neural networks (SNNs) by estimating the approximate gradients of spiking activity through a continuous function with constant sharpness, known as surrogate gradient (SG) learning. However, as spikes propagate within neurons and among layers, the distribution of membrane potential dynamics (MPD) will deviate from the gradient-available interval of fixed SG, hindering SNNs from searching the optimal solution space. To maintain the stability of gradient flows, SG needs to align with evolving MPD. Here, we propose a novel adaptive gradient learning for SNNs by exploiting MPD, namely MPD-AGL. It fully accounts for the underlying factors contributing to membrane potential shifts and establishes a dynamic association between SG and MPD at different timesteps to relax gradient estimation, which provides a new degree of freedom for SG learning. Experimental results demonstrate that our method achieves excellent performance at low latency. Moreover, it increases the proportion of neurons that fall into the gradient-available interval compared to fixed SG, effectively mitigating the gradient vanishing problem. Code is available at https://github.com/jqjiang1999/MPD-AGL.

Video surveillance is widely used in various public spaces, and person re-identification (ReId) based on video frames has become a research hotspot in the field of computer vision. However, most video-based methods struggle with issues such as poor lighting or motion blur, which result in blurry textures and hinder the extraction of discriminative features for specific identities. Additionally, frame-based methods may introduce privacy concerns and suffer from redundant storage between frames. In contrast, bio-inspired sensors—such as event cameras—have asynchronous properties and higher temporal resolution. They record events when changes in light intensity occur, offering a better response to lighting variations in specific scenes. Spiking Neural Networks (SNNs), which transmit information through sparse spikes, are naturally suited to process such asynchronous and sparse event-stream inputs. In this work, we explore the possibility of using only SNN for person ReId based on event-stream data generated by event cameras for the first time. While SNNs for person ReId suffer from issues such as blurry matching and poor feature distinction in spike-based feature matching, we propose an adaptive temporal feature representation (ATFR) method based on membrane potential to address these issues. We evaluate our method on two event-based datasets (Event-ReId and Event-PRID-2011). Compared to existing methods, our model achieves competitive performance while improving energy efficiency by at least 6 times.

Spiking neural networks (SNNs) operating with asynchronous discrete events show higher energy efficiency with sparse computation. A popular approach for implementing deep SNNs is artificial neural network (ANN)-SNN conversion combining both efficient training of ANNs and efficient inference of SNNs. However, the accuracy loss is usually nonnegligible, especially under few time steps, which restricts the applications of SNN on latency-sensitive edge devices greatly. In this article, we first identify that such performance degradation stems from the misrepresentation of the negative or overflow residual membrane potential in SNNs. Inspired by this, we decompose the conversion error into three parts: quantization error, clipping error, and residual membrane potential representation error. With such insights, we propose a two-stage conversion algorithm to minimize those errors, respectively. In addition, we show that each stage achieves significant performance gains in a complementary manner. By evaluating on challenging datasets including CIFAR- 10, CIFAR- 100, and ImageNet, the proposed method demonstrates the state-of-the-art performance in terms of accuracy, latency, and energy preservation. Furthermore, our method is evaluated using a more challenging object detection task, revealing notable gains in regression performance under ultralow latency, when compared with existing spike-based detection algorithms. Codes will be available at: https://github.com/Windere/snn-cvt-dual-phase.

Existing approaches usually perform spatiotemporal representation in the spatial and temporal dimensions, respectively, which isolates the spatial and temporal natures of the target and leads to sub-optimal embeddings. Neuroscience research has shown that the mammalian brain entorhinal-hippocampal system provides efficient graph representations for general knowledge. Moreover, entorhinal grid cells present concise spatial representations, while hippocampal place cells represent perception conjunctions effectively. Thus, the entorhinal-hippocampal system provides a novel angle for spatiotemporal representation, which inspires us to propose the SpatioTemporal aware Embedding framework (STE) and apply it to POIs (STEP). STEP considers two types of POI-specific representations: sequential representation and spatiotemporal conjunctive representation, learned using sparse unlabeled data based on the proposed graph-building policies. Notably, STEP jointly represents the spatiotemporal natures of POIs using both observations and contextual information from integrated spatiotemporal dimensions by constructing a spatiotemporal context graph. Furthermore, we introduce a successive POI recommendation method using STEP, which achieves state-of-the-art performance on two benchmarks. In addition, we demonstrate the excellent performance of the STE representation approach in other spatiotemporal representation-centered tasks through a case study of the traffic flow prediction problem. Therefore, this work provides a novel solution to spatiotemporal representation and paves a new way for spatiotemporal modeling-related tasks.

Automated service classification plays a pivotal role in the processes of service discovery, selection, and composition. In real scenarios, service classification usually faces an open world where out-of-distribution (OOD) service samples exist and cannot be fully covered by a predefined class label set. This problem arises when either new services (mobile applications, protocols, etc.) appear or known services change their behaviour. Hence, the task of OOD detection has been attracting more and more attention in recent years. In this paper, based on feature representations with supervised contrastive learning, we propose a novel joint learning model that can estimate whether an input service sample is OOD and in the meantime, predict its in-distribution (IND) class label. Moreover, we propose to synthesize OOD service samples with known IND service samples from different classes, which can further help in training the proposed model. The extensive experimental evaluation on three datasets shows that our method outperforms the state-of-the-art model in most cases, and can obtain a significant increase in Accuracy and F1-score over the unknown class when the proportion of IND classes is low.

Event-based cameras provide a unique way of visual perception through the event-driven characteristics and representation of sparse spatiotemporal data. Meanwhile, as brain-inspired models, Spiking Neural Networks (SNNs) have asynchronous event processing characteristics and are able to process event data naturally. However, various feature extraction methods currently in SNNs have not fully utilized the temporal information in the output of bionic visual sensors, making it difficult to better maintain Address Event Representations (AER). In this paper, we proposed a method to enhance time surfaces by extracting hidden temporal information in the time surface and multi-scale dilated time surface. The method of extracting hidden temporal information (EHTI) is to enhance the time surface by exploiting the information of time surface changes between the current event and other events in the spatiotemporal neighborhood, allowing the event’s temporal features to be fully utilized. In addition, we adopted the multi-scale dilated time surface (MDTS) method to solve the problem of limited event receptive fields by using dilated time surfaces for multi-scale time surface feature extraction, which can better capture the spatiotemporal connections between events in further neighborhoods. Experimental results on various event-based datasets (i.e., N-MNIST, MNIST-DVS, DVS128 Gesture, and DailyAction-DVS) show that our method is able to extract richer features and outperforms other methods on classification tasks.

In the cognitive function of the brain, memory plays a crucial role, which involves a complex process of encoding, storing, and retrieving information. The key mechanism in this process is synaptic plasticity, which allows excitatory and inhibitory neural circuits in the brain to adjust the strength of synapses based on experience and learning, facilitating the storage of information. In this study, we designed an associative memory model using spiking neural networks (SNNs) with spike-timing-dependent plasticity (STDP), aiming to simulate the excitatory and inhibitory neural circuits in the brain responsible for encoding, storing, and retrieving memories. In this model, two groups of excitatory neurons and a group of inhibitory neurons in the memory layer receive inputs and activate. Synaptic states between these neurons are modified through STDP during this process, including strengthening, weakening, or forming new synapses related to memory. Subsequently, through the generated synapses, cues guide the activation of one group of excitatory neurons in the memory layer and then trigger the response of another group of excitatory neurons, thereby achieving memory retrieval, namely recall. The results show that our model successfully achieved memory storage and retrieval in auto-associative and hetero-associative tasks. We also discussed the relationship between synaptic utilization of the model and the number of input patterns. Finally, we verified the role of inhibitory neurons in maintaining the stability of the memory network.

Event streams output by event cameras have low data redundancy and retain accurate temporal information in the form of Address Event Representation (AER) which are different from the outputs of traditional frame-based cameras. Spiking Neural Networks (SNNs) are considered an effective tool for handling event-based scenarios due to their inherent temporal properties. However, most existing SNNs directly convert an event stream to several static frames with temporal relationships by channel-wise accumulation of events. These serial frames lose temporal characteristics in some extent and potentially affect the capacity of the SNNs to learn and recognize event streams. In this work, we proposed a novel event-based feature descriptor called time interval correlation time-surface (TICTS) for SNNs and introduced this event-based feature extraction method into SNNs. The TICTS can capture more precise temporal correlation from event streams, thereby facilitating SNNs to learn temporal information more effectively. The experimental results show that SNNs with the proposed TICTS exhibit superior performance and increased stability across datasets with varying speeds. In addition, shallow network using TICTS can achieve competitive accuracy compared to deep networks, underscoring the effectiveness of TICTS in reducing the redundant network size of SNNs.

Event cameras have attracted increasing attention in the field of computer vision due to their advantages in terms of high temporal resolution, high dynamic range and low power consumption. However, the output of event cameras is a sparse and discrete event stream, with each individual event in the event streams containing only little information. Therefore, extracting more effective features from the available information in the event streams is currently a major challenge in event-based object classification. In this paper, we propose a novel event-based feature representation to encode the spatiotemporal features of event streams. The Harmonic Mean Time Surfaces (HMTS) representation makes efficient use of information about past events, which enhances the spatiotemporal relationship between events, thus establishing robust representation. Furthermore, we propose a multi-scale feature extraction model that can construct broader event-region correlations. In the classification stage, the extracted spatiotemporal features are classified using a spiking neural network with an event-driven Tempotron rule. To demonstrate the effectiveness of the proposed method, we conduct a series of experiments on the N-MNIST, MNIST-DVS, DVS128 Gesture and DailyAction-DVS datasets. Experimental results show that our model achieves superior classification performance and exhibits a high level of robustness to noise.

Developing an efficient collision avoidance policy with Spiking Reinforcement Learning for dynamic, decision-making agents remains challenging. Moreover, the implementation of energy-efficient collision avoidance is important for mobile robots that operate with limited on-board computing resources. Most existing energy-efficient methods via spiking reinforcement learning are predominately concerned with the navigational capabilities of a single agent, and are unable to handle a large, and possibly varying number of agents. To overcome these limitations, we propose a model called collision avoidance with spiking reinforcement learning (CASRL), based on proximal policy optimization algorithms. This proposed model consists of an actor with spiking neural networks (SNNs) and a critic with deep neural networks (DNNs). Our spiking reinforcement learning algorithm is advantageous to handle an arbitrary number of other agents by virtue of a spiking-gated transformer (SpikeGTr) architecture and an accumulate-to-fire (ATF) module. Extensive experimental results demonstrate that CASRL obtains a competitive success rate of navigation and exhibits higher time-efficiency for navigation in crowded scenarios compared to traditional DNN-based methods.

While supervised learning of over-parameterized neural networks achieved state-of-the-art performance in image classification, it tends to over-fit the labeled training samples to give inferior generalization ability. Output regularization deals with over-fitting by using soft targets as additional training signals. Although clustering is one of the most fundamental data analysis tools for discovering general-purpose and data-driven structures, it has been ignored in existing output regularization approaches. In this article, we leverage this underlying structural information by proposing Cluster-based soft targets for Output Regularization (CluOReg). This approach provides a unified way for simultaneous clustering in embedding space and neural classifier training with cluster-based soft targets via output regularization. By explicitly calculating a class relationship matrix in the cluster space, we obtain classwise soft targets shared by all samples in each class. Results of image classification experiments under various settings on a number of benchmark datasets are provided. Without resorting to external models or designed data augmentation, we get consistent and significant reductions in classification error compared with other approaches, demonstrating that cluster-based soft targets effectively complement the ground-truth label.

The temporal credit assignment (TCA) problem, which aims to detect predictive features hidden in distracting background streams, remains a core challenge in biological and machine learning. Aggregate-label (AL) learning is proposed by researchers to resolve this problem by matching spikes with delayed feedback. However, the existing AL learning algorithms only consider the information of a single timestep, which is inconsistent with the real situation. Meanwhile, there is no quantitative evaluation method for TCA problems. To address these limitations, we propose a novel attention-based TCA (ATCA) algorithm and a minimum editing distance (MED)-based quantitative evaluation method. Specifically, we define a loss function based on the attention mechanism to deal with the information contained within the spike clusters and use MED to evaluate the similarity between the spike train and the target clue flow. Experimental results on musical instrument recognition (MedleyDB), speech recognition (TIDIGITS), and gesture recognition (DVS128-Gesture) show that the ATCA algorithm can reach the state-of-the-art (SOTA) level compared with other AL learning algorithms.

Spiking neural networks (SNNs) with biologically inspired spatio-temporal dynamics demonstrate superior energy efficiency on neuromorphic architectures. Error backpropagation in SNNs is prohibited by the all-or-none nature of spikes. The existing solution circumvents this problem by a relaxation on the gradient calculation using a continuous function with a constant relaxation de- gree, so-called surrogate gradient learning. Nevertheless, such a solution introduces additional smoothing error on spike firing which leads to the gradients being estimated inaccurately. Thus, how to adaptively adjust the relaxation degree and eliminate smoothing error progressively is crucial. Here, we propose a methodology such that training a prototype neural network will evolve into training an SNN gradually by fusing the learnable relaxation degree into the network with random spike noise. In this way, the network learns adaptively the accurate gradients of loss landscape in SNNs. The theoretical analysis further shows optimization on such a noisy network could be evolved into optimization on the embedded SNN with shared weights progressively. Moreover, The experiments on static images, dynamic event streams, speech, and instrumental sounds show the proposed method achieves state-of-the-art performance across all the datasets with remarkable robustness on different relaxation degrees.

Spiking neural networks (SNNs) have increasingly drawn massive research attention due to biological interpretability and efficient computation. Recent achievements are devoted to utilizing the surrogate gradient (SG) method to avoid the dilemma of non-differentiability of spiking activity to directly train SNNs by backpropagation. However, the fixed width of the SG leads to gradient vanishing and mismatch problems, thus limiting the performance of directly trained SNNs. In this work, we propose a novel perspective to unlock the width limitation of SG, called the learnable surrogate gradient (LSG) method. The LSG method modulates the width of SG according to the change of the distribution of the membrane potentials, which is identified to be related to the decay factors based on our theoretical analysis. Then we introduce the trainable decay factors to implement the LSG method, which can optimize the width of SG automatically during training to avoid the gradient vanishing and mismatch problems caused by the limited width of SG. We evaluate the proposed LSG method on both image and neuromorphic datasets. Experimental results show that the LSG method can effectively alleviate the blocking of gradient propagation caused by the limited width of SG when training deep SNNs directly. Meanwhile, the LSG method can help SNNs achieve competitive performance on both latency and accuracy.

In recent years, spiking neural networks (SNNs) have been used in reinforcement learning (RL) due to their low power consumption and event-driven features. However, spiking reinforcement learning (SRL), which suffers from fixed coding methods, still faces the problems of high latency and poor versatility. In this paper, we use learnable matrix multiplication to encode and decode spikes, improving the flexibility of the coders and thus reducing latency. Meanwhile, we train the SNNs using the direct training method and use two different structures for online and offline RL algorithms, which gives our model a wider range of applications. Extensive experiments have revealed that our method achieves optimal performance with ultra-low latency (as low as 0.8% of other SRL methods) and excellent energy efficiency (up to 5X the DNNs) in different algorithms and different environments.

Developing computational models of neural response is crucial for understanding sensory processing and neural computations. Current state-of-the-art neural network methods use temporal filters to handle temporal dependencies, resulting in an **unrealistic and inflexible processing paradigm**. Meanwhile, these methods target **trial-averaged firing rates** and fail to capture important features in spike trains. This work presents the temporal conditioning spiking latent variable models (***TeCoS-LVM***) to simulate the neural response to natural visual stimuli. We use spiking neurons to produce spike outputs that directly match the recorded trains. This approach helps to avoid losing information embedded in the original spike trains. We exclude the temporal dimension from the model parameter space and introduce a temporal conditioning operation to allow the model to adaptively explore and exploit temporal dependencies in stimuli sequences in a **natural paradigm**. We show that TeCoS-LVM models can produce more realistic spike activities and accurately fit spike statistics than powerful alternatives. Additionally, learned TeCoS-LVM models can generalize well to longer time scales. Overall, while remaining computationally tractable, our model effectively captures key features of neural coding systems. It thus provides a useful tool for building accurate predictive computational accounts for various sensory perception circuits.

Backpropagation has been successfully generalized to optimize deep spiking neural networks (SNNs), where, nevertheless, gradients need to be propagated back through all layers, resulting in a massive consumption of computing resources and an obstacle to the parallelization of training. A biologically motivated scheme of local learning provides an alternative to efficiently train deep networks but often suffers a low performance of accuracy on practical tasks. Thus, how to train deep SNNs with the local learning scheme to achieve both efficient and accurate performance still remains an important challenge. In this study, we focus on a supervised local learning scheme where each layer is independently optimized with an auxiliary classifier. Accordingly, we first propose a spike-based efficient local learning rule by only considering the direct dependencies in the current time. We then propose two variants that additionally incorporate temporal dependencies through a backward and forward process, respectively. The effectiveness and performance of our proposed methods are extensively evaluated with six mainstream datasets. Experimental results show that our methods can successfully scale up to large networks and substantially outperform the spike-based local learning baselines on all studied benchmarks. Our results also reveal that gradients with temporal dependencies are essential for high performance on temporal tasks, while they have negligible effects on rate-based tasks. Our work is significant as it brings the performance of spike-based local learning to a new level with the computational benefits being retained.

Recent work on spiking neural networks (SNNs) has focused on achieving deep architectures. They commonly use backpropagation (BP) to train SNNs directly, which allows SNNs to go deeper and achieve higher performance. However, the BP training procedure is computing intensive and complicated by many trainable parameters. Inspired by global pooling in convolutional neural networks (CNNs), we present the spike probabilistic global pooling (SPGP) method based on a probability function for training deep convolutional SNNs. It aims to remove the difficulty of too many trainable parameters brought by multiple layers in the training process, which can reduce the risk of overfitting and get better performance for deep SNNs (DSNNs). We use the discrete leaky-integrate-fire model and the spatiotemporal BP algorithm for training DSNNs directly. As a result, our model trained with the SPGP method achieves competitive performance compared to the existing DSNNs on image and neuromorphic data sets while minimizing the number of trainable parameters. In addition, the proposed SPGP method shows its effectiveness in performance improvement, convergence, and generalization ability.

Learning new concepts rapidly from a few examples is an open issue in spike-based machine learning. This few-shot learning imposes substantial challenges to the current learning methodologies of spiking neuron networks (SNNs) due to the lack of task-related priori knowledge. The recent learning-to-learn (L2L) approach allows SNNs to acquire priori knowledge through example-level learning and task-level optimization. However, existing L2L-based frameworks do not target the neural dynamics (i.e., neuronal and synaptic parameter changes) on different timescales. This diversity of temporal dynamics is an important attribute in spike-based learning, which facilitates the networks to rapidly acquire knowledge from very few examples and gradually integrate this knowledge. In this work, we consider the neural dynamics on various timescales and provide a multi-timescale optimization (MTSO) framework for SNNs. This framework introduces an adaptive-gated LSTM to accommodate two different timescales of neural dynamics: short-term learning and long-term evolution. Short-term learning is a fast knowledge acquisition process achieved by a novel surrogate gradient online learning (SGOL) algorithm, where the LSTM guides gradient updating of SNN on a short timescale through an adaptive learning rate and weight decay gating. The long-term evolution aims to slowly integrate acquired knowledge and form a priori, which can be achieved by optimizing the LSTM guidance process to tune SNN parameters on a long timescale. Experimental results demonstrate that the collaborative optimization of multi-timescale neural dynamics can make SNNs achieve promising performance for the few-shot learning tasks.

In recent years, the learning of deep spiking neural networks(SNN) has attracted increasing researchers' interest, and has also made important progresses in theories and applications. It is desired to choose a neuron model with biological features and suitable for SNN training. Currently, Leaky Integrate-and-Fire(LIF) model is mainly used in deep SNN and some factors that can express the spatio-temporal information are ignored in the model. In this work, inspired by the Hodgkin-Huxley(H-H) model, we propose an improved LIF neuron model, which is an iterative current-based LIF model with voltage-based variable resistance. The improved neuron model is closer to the characteristics of the biological neuron model, which can make use of the spatio-temporal information. We further construct a new SNN learning algorithm that uses spatio-temporal back propagation by defining a loss function. We evaluated the proposed methods on single-label and multi-label data sets. The experimental results show that the variable resistance of the neuron model will affect the performance of the model. Choosing the appropriate relationship between the variable resistance and the membrane voltage can effectively improve the recognition accuracy.

A spiking neural network (SNN) is a type of biological plausibility model that performs information processing based on spikes. Training a deep SNN effectively is challenging due to the nondifferention of spike signals. Recent advances have shown that high-performance SNNs can be obtained by converting convolutional neural networks (CNNs). However, the large-scale SNNs are poorly served by conventional architectures due to the dynamic nature of spiking neurons. In this letter, we propose a hardware architecture to enable efficient implementation of SNNs. All layers in the network are mapped on one chip so that the computation of different time steps can be done in parallel to reduce latency. We propose new spiking max-pooling method to reduce computation complexity. In addition, we apply approaches based on shift register and coarsely grained parallels to accelerate convolution operation. We also investigate the effect of different encoding methods on SNN accuracy. Finally, we validate the hardware architecture on the Xilinx Zynq ZCU102. The experimental results on the MNIST data set show that it can achieve an accuracy of 98.94% with eight-bit quantized weights. Furthermore, it achieves 164 frames per second (FPS) under 150 MHz clock frequency and obtains 41[Formula: see text] speed-up compared to CPU implementation and 22 times lower power than GPU implementation.

In this article, we present a systematic computational model to explore brain-based computation for object recognition. The model extracts temporal features embedded in address-event representation (AER) data and discriminates different objects by using spiking neural networks (SNNs). We use multispike encoding to extract temporal features contained in the AER data. These temporal patterns are then learned through the tempotron learning rule. The presented model is consistently implemented in a temporal learning framework, where the precise timing of spikes is considered in the feature-encoding and learning process. A noise-reduction method is also proposed by calculating the correlation of an event with the surrounding spatial neighborhood based on the recently proposed time-surface technique. The model evaluated on wide spectrum data sets (MNIST, N-MNIST, MNIST-DVS, AER Posture, and Poker Card) demonstrates its superior recognition performance, especially for the events with noise.

The formulation of efficient supervised learning algorithms for spiking neurons is complicated and remains challenging. Most existing learning methods with the precisely firing times of spikes often result in relatively low efficiency and poor robustness to noise. To address these limitations, we propose a simple and effective multi-spike learning rule to train neurons to match their output spike number with a desired one. The proposed method will quickly find a local maximum value (directly related to the embedded feature) as the relevant signal for synaptic updates based on membrane potential trace of a neuron, and constructs an error function defined as the difference between the local maximum membrane potential and the firing threshold. With the presented rule, a single neuron can be trained to learn multi-category tasks, and can successfully mitigate the impact of the input noise and discover embedded features. Experimental results show the proposed algorithm has higher precision, lower computation cost, and better noise robustness than current state-of-the-art learning methods under a wide range of learning tasks.

Temporal encoding of visual stimulus based on the spiking neural networks is important and challenging. Inspired by the representation of neuron population in the sensory pathway, we propose a new hierarchical encoding method, which amplifies the difference among spike trains through the synaptic projection matrix and leaves the top neurons keeping active by a winner-take-all scheme. Finally, the neurons are assigned to positive and negative subthreshold membrane oscillation to fire new spike trains. In the optical character recognition task, we compare the readout performance of the proposed encoding method and the phase encoding method. When the image is severely destroyed, for example, at 25% noise, our method still maintains 88% recognition accuracy, which is 30% higher than the accuracy of phase encoding. The robustness of readout neurons to perform recognition tasks is enhanced through the proposed encoding method. We performed the benchmark two-class classification experiment on Caltech 101 dataset, and the accuracy can reach 99.4%, which is 17% higher than the phase encoding. Compared with the other spiking deep neural network, our method also shows great improvement and strong scalability.

Though succeeding in solving various learning tasks, most existing reinforcement learning (RL) models have failed to take into account the complexity of synaptic plasticity in the neural system. Models implementing reinforcement learning with spiking neurons involve only a single plasticity mechanism. Here, we propose a neural realistic reinforcement learning model that coordinates the plasticities of two types of synapses: stochastic and deterministic. The plasticity of the stochastic synapse is achieved by the hedonistic rule through modulating the release probability of synaptic neurotransmitter, while the plasticity of the deterministic synapse is achieved by a variant of a reward-modulated spike-timing-dependent plasticity rule through modulating the synaptic strengths. We evaluate the proposed learning model on two benchmark tasks: learning a logic gate function and the 19-state random walk problem. Experimental results show that the coordination of diverse synaptic plasticities can make the RL model learn in a rapid and stable form.

Robust object recognition in spiking neural systems remains a challenging in neuromorphic computing area as it needs to solve both the effective encoding of sensory information and also its integration with downstream learning neurons. We target this problem by developing a spiking neural system consisting of sparse temporal encoding and temporal classifier. We propose a sparse temporal encoding algorithm which exploits both spatial and temporal information derived from an spike-timing-dependent plasticity-based HMAX feature extraction process. The temporal feature representation, thus, becomes more appropriate to be integrated with a temporal classifier based on spiking neurons rather than with nontemporal classifier. The algorithm has been validated on two benchmark data sets and the results show the temporal feature encoding and learning-based method achieves high recognition accuracy. The proposed model provides an efficient approach to perform feature representation and recognition in a consistent temporal learning framework, which is easily adapted to neuromorphic implementations.

In this paper, we address two challenging problems in unsupervised subspace learning: 1) how to automatically identify the feature dimension of the learned subspace (i.e., automatic subspace learning) and 2) how to learn the underlying subspace in the presence of Gaussian noise (i.e., robust subspace learning). We show that these two problems can be simultaneously solved by proposing a new method [(called principal coefficients embedding (PCE)]. For a given data set , PCE recovers a clean data set from and simultaneously learns a global reconstruction relation of . By preserving into an -dimensional space, the proposed method obtains a projection matrix that can capture the latent manifold structure of , where is automatically determined by the rank of with theoretical guarantees. PCE has three advantages: 1) it can automatically determine the feature dimension even though data are sampled from a union of multiple linear subspaces in presence of the Gaussian noise; 2) although the objective function of PCE only considers the Gaussian noise, experimental results show that it is robust to the non-Gaussian noise (e.g., random pixel corruption) and real disguises; and 3) our method has a closed-form solution and can be calculated very fast. Extensive experimental results show the superiority of PCE on a range of databases with respect to the classification accuracy, robustness, and efficiency.

Address event representation (AER) image sensors represent the visual information as a sequence of events that denotes the luminance changes of the scene. In this paper, we introduce a feature extraction method for AER image sensors based on the probability theory, namely, bag of events (BOE). The proposed approach represents each object as the joint probability distribution of the concurrent events, and each event corresponds to a unique activated pixel of the AER sensor. The advantages of BOE include: 1) it is a statistical learning method and has a good interpretability in mathematics; 2) BOE can significantly reduce the effort to tune parameters for different data sets, because it only has one hyperparameter and is robust to the value of the parameter; 3) BOE is an online learning algorithm, which does not require the training data to be collected in advance; 4) BOE can achieve competitive results in real time for feature extraction (>275 frames/s and >120,000 events/s); and 5) the implementation complexity of BOE only involves some basic operations, e.g., addition and multiplication. This guarantees the hardware friendliness of our method. The experimental results on three popular AER databases (i.e., MNIST-dynamic vision sensor, Poker Card, and Posture) show that our method is remarkably faster than two recently proposed AER categorization systems while preserving a good classification accuracy.

Parallel with the development of service robots, it is vital for the robots to carry out handovers autonomously. Robot-to-human handover is a coordination in time and space for a robot to deliver an object to human. A good robot-to-human handover should consider human safety and preference, natural motion planning that mimics human and adaptability to the changes of the environment. Conventional handover motion mostly rely on sampling-based algorithms that emphasizes on kinematic and dynamic analysis. This kind of motion planning could become complicated and slow in response if the handover motion is implemented in a dynamic environment where real time motion planning is required. To simplify the implementation of robot-to-human handover, a motion learning and generation framework that based on Continuous Time Recurrent Neural Network(CTRNN) is proposed. The proposed framework is equipped with the capabilities of object recognition, motion generation based on past learning experience and obstacle adaptation. As compared with conventional method, the proposed framework could be easily extended to handover motion with high dimensional configuration spaces as the motion can be generated from the learnt experience. In the proposed framework, the handover behaviour can be learnt via human-guided motion teaching which provides an intuitive and visible solution for motion planning. The proposed framework has been experimentally evaluated on a customized design robot via robotto-human handover testing. Based on the testing, the feasibility of the proposed framework had been justified.

This paper proposes a biologically plausible network architecture with spiking neurons for sequence recognition. This architecture is a unified and consistent system with functional parts of sensory encoding, learning, and decoding. This is the first systematic model attempting to reveal the neural mechanisms considering both the upstream and the downstream neurons together. The whole system is a consistent temporal framework, where the precise timing of spikes is employed for information processing and cognitive computing. Experimental results show that the system is competent to perform the sequence recognition, being robust to noisy sensory inputs and invariant to changes in the intervals between input stimuli within a certain range. The classification ability of the temporal learning rule used in the system is investigated through two benchmark tasks that outperform the other two widely used learning rules for classification. The results also demonstrate the computational power of spiking neurons over perceptrons for processing spatiotemporal patterns. In summary, the system provides a general way with spiking neurons to encode external stimuli into spatiotemporal spikes, to learn the encoded spike patterns with temporal learning rules, and to decode the sequence order with downstream neurons. The system structure would be beneficial for developments in both hardware and software.

In this paper, we propose a framework to analyze the interactive behaviors of humans and robots in physical interactions. Game theory is employed to describe the system under study, and policy iteration is adopted to provide a solution of Nash equilibrium. The human's control objective is estimated based on the measured interaction force, and it is used to adapt the robot's objective such that human-robot coordination can be achieved. The validity of the proposed method is verified through a rigorous proof and experimental studies.

In this paper, a role adaptation method is developed for human-robot collaboration based on game theory. This role adaptation is engaged whenever the interaction force changes, causing the proportion of control sharing between human and robot to vary. In one boundary condition, the robot takes full control of the system when there is no human intervention. In the other boundary condition, it becomes a follower when the human exhibits strong intention to lead the task. Experimental results show that the proposed method yields better overall performance than fixed-role interactions.

In this paper, we propose an adaptive optimal control for a robot to collaborate with a human. Game theory and policy iteration are employed to analyze the interactive behaviors of the human and the robot in physical interactions. The human's control objective is estimated and it is used to adapt the robot's own objective, such that human-robot coordination can be achieved. An optimal control is developed to guarantee that the robot's control objective is realized. The validity of the proposed method is verified through rigorous analysis and experiment studies.

In this paper, we propose a role adaptation method for human-robot shared control. Game theory is employed for fundamental analysis of this two-agent system. An adaptation law is developed such that the robot is able to adjust its own role according to the human's intention to lead or follow, which is inferred through the measured interaction force. In the absence of human interaction forces, the adaptive scheme allows the robot to take the lead and complete the task by itself. On the other hand, when the human persistently exerts strong forces that signal an unambiguous intent to lead, the robot yields and becomes the follower. Additionally, the full spectrum of mixed roles between these extreme scenarios is afforded by continuous online update of the control that is shared between both agents. Theoretical analysis shows that the resulting shared control is optimal with respect to a two-agent coordination game. Experimental results illustrate better overall performance, in terms of both error and effort, compared with fixed-role interactions.

It is undeniable that the ability to grasp and handle an object is vital for service robots. From object recognition to object grasping motion, the motion execution should be as fast as possible. Due to the possible position variation of the target object to be grasped, online planning of grasping motion should be done. In order to achieve flexible grasping motion, recurrent neural network could be implemented as an alternative to conventional manipulation method which is based on kinematic and dynamic analysis. However, the application of recurrent neural network model requires good and easily obtainable training data. Hence, a novel robotic arm design with high flexibility is proposed to facilitate the training and implementation of the recurrent neural network model. The feasibility of the proposed robotic arm design is evaluated via the training, learning and testing of stochastic continuous time recurrent neural network (S-CTRNN) model with grasping a box motion.

The application of robotics to telepresence can enhance user interaction experience by providing embodiment, engaging behaviors, automatic control, and human perception. This paper presents a new telepresence robot with gesture-based attention direction to orient the robot towards attention targets according to human deictic gestures. Gesture-based attention direction is realized by combining Localist Attractor Network (LAN) and Short-Term Memory (STM).We also propose audio-visual fusion based on context-dependent prioritization among the 3 types of audio-visual cues (gesture, speech source location, head location). Experiment results are very promising and show that i) the average gesture recognition rate is 92%, i) gesture-based attention direction rate is 90%, and that ii) only by considering the 3 types of audio-visual cues together can the robot perform on par with a human in directing attention to the correct person in a meeting scenario.

A singularity-robust inverse kinematics framework with self-collision avoidance is proposed for a 7 degree-of-freedom (DOF) robot arm, based on minimization of energy consumption. We consider a fully revolute and redundant robot arm, consisting of two spherical joints located at the shoulder and the wrist, connected by a hinge joint at the elbow. This kinematic configuration allows the elbow to swivel freely about an axis joining the wrist and shoulder, thus allowing the redundancy to be parameterized by a single variable, namely the swivel angle. Closed form solutions for the inverse kinematics (IK) problem exist if the elbow position is known. Generally, a set of valid IK solutions, which comply with structural constraints, can be obtained from the entire range of solutions that are generated by swiveling the elbow through 360°. An objective function is proposed to determine the optimal joint trajectory based on a minimum energy criterion. To complete the framework, the issue of kinematic singularity is handled by using the concept of energy minimization.

Motivated by applications in robot-assisted physical rehabilitation, this paper presents an adaptive control design for robot manipulators operating in an ellipsoidal constrained region. The ellipsoidal constraint problem is more challenging than the box constraint problem tackled in previous works, since the nonlinear constraint boundary cannot be handled in a decoupled manner along the dimensions of the task space. We introduce a novel Barrier Lyapunov Function (BLF) which contains a quotient of the squared norm of the tracking error over the ellipsoidal task space constraint. This function allows the task space constraint to be handled directly without requiring an intermediate mapping to the error space. We show that, under the proposed BLF-based adaptive control, the end-effector always remains in the constrained region despite the perturbing effects of online parameter adaptation and also the presence of bounded external disturbances. A simulation example illustrates the performance of the proposed control.

We present adaptive admittance control of a robotic manipulator, with uncertain dynamic parameters, operating in a constrained task space. To provide compliance to external forces, we generate a differentiable reference trajectory that remains in the constrained task space. Then, adaptive backstepping control, based on a time-varying asymmetric Barrier Lyapunov Function (BLF), is designed to achieve tracking of the reference trajectory while guaranteeing constraint satisfaction. The improved BLF-based control renders the entire constrained task space positively invariant. Despite transient perturbations by external forces and online parameter adaptation, practical tracking of the reference trajectory is achieved without transgression of the constrained task space. In the absence of interaction forces, asymptotic tracking of the desired trajectory is achieved.

Memory is a fundamental part of computational systems like the human brain. Theoretical models identify memories as attractors of neural network activity patterns based on the theory that attractor (recurrent) neural networks are able to capture some crucial characteristics of memory, such as encoding, storage, retrieval, and long-term and working memory. In such networks, long-term storage of the memory patterns is enabled by synaptic strengths that are adjusted according to some activity-dependent plasticity mechanisms (of which the most widely recognized is the Hebbian rule) such that the attractors of the network dynamics represent the stored memories. Most of previous studies on associative memory are focused on Hopfield-like binary networks, and the learned patterns are often assumed to be uncorrelated in a way that minimal interactions between memories are facilitated. In this letter, we restrict our attention to a more biological plausible attractor network model and study the neuronal representations of correlated patterns. We have examined the role of saliency weights in memory dynamics. Our results demonstrate that the retrieval process of the memorized patterns is characterized by the saliency distribution, which affects the landscape of the attractors. We have established the conditions that the network state converges to unique memory and multiple memories. The analytical result also holds for other cases for variable coding levels and nonbinary levels, indicating a general property emerging from correlated memories. Our results confirmed the advantage of computing with graded-response neurons over binary neurons (i.e., reducing of spurious states). It was also found that the nonuniform saliency distribution can contribute to disappearance of spurious states when they exit.

In this paper, a learning control approach is applied to the synchronization of two uncertain chaotic systems which contain nonlinear uncertainties with unknown time delays. This learning approach also deals with unknown time-varying parameters having distinct periods in the master and slave systems. Using the Lyapunov-Krasovskii functional and incorporating periodic parametric learning mechanism, global stability and asymptotic synchronization between the master and the slave systems are obtained. Simulation studies on representative classes of chaotic systems demonstrate the effectiveness of the proposed approach