In this paper, we propose the Schoenberg kernel loss, a loss function that measures the divergence between spike trains for spiking neural network training. Our method addresses the challenges of non-differentiable spiking neurons and leverages both spatial and temporal dynamics in information processing. We evaluated our approach on the neuromorphic benchmark datasets, N-MNIST, and DVS Gesture, demonstrating our method outperforms previously published loss functions on our network architecture. In addition, we extend our investigation to real-time classification simulations in real-world scenarios. This work contributes to the advancement of direct SNN training methods, offering improved performance, particularly on low spike density data, and potential applications in neuromorphic classification.

We propose a novel cascaded classification technique called the Nearest Instance Centroid Estimation (NICE) LDA algorithm. Our algorithm (inspired from NICE KLMS) performs a cascade combination of two weak classifiers - threshold based class-wise clustering and linear discriminant classification to achieve state-of-the-art results on various high dimensional UCI datasets. We show how our method is more robust towards skewed data and computationally more efficient than previous methods of combining clustering with classification techniques. We also develop an efficient aggregation method based on instance based learning that implements this cascade combination of classifiers in a much simpler manner computationally. We demonstrate that our method of data clustering and LDA implementation, while introducing only one free parameter, leads to results that are similar and often better than those achieved by the state-of-the-art kernel RBF SVMs.

In this paper, we propose a novel, combined surprise-novelty approach to online active learning for kernel adaptive filters. Surprise and novelty criteria have always been used individually in designing sparse kernel machines. While closely related, they are in fact two distinct concepts. In this paper, we highlight the key differences between surprise and novelty, define quantitative measures, and propose a unifying framework that leverages the complementary properties of the two concepts combined. We test the information theoretic approach of designing sparse kernel adaptive filters using the surprise-novelty information processing for Gaussian online active learning (SNIP-GOAL) on the task of nonlinear chaotic time series prediction. The proposed method outperforms existing algorithms using the measures individually. Results show that combining surprise and novelty can be advantageous in terms of efficiency and performance. Leveraging both measures allows the system to be not only sparse but also generalizes better.

Automatic insect recognition (AIR), using noninvasive methods in situ, has far-reaching implications in entomology, agriculture, and disease control and prevention. An emerging technology in computational entomology uses flight information captured by laser sensors. Current methods treat these optical signals as static patterns, rather than time series. We propose a novel approach to AIR by evaluating each insect passage as a nonstationary process involving a sequence of pseudo-acoustic frames and modeling the short-term flight dynamics using the kernel adaptive autoregressive-moving average (KAARMA) algorithm. Since flight behavior is both nonlinear and nonstationary in nature, dynamic modeling provides a general framework that fully exploits the transitional and contextual information. Results show KAARMA classifier outperforms the state-of-the-art AIR methods, using support vector machine (SVM), deep-learning autoencoder, and batch learning, in identifying Zika vector mosquito Aedes aegypti among five species of flying insects, while using significantly more efficient data representation.

In this paper, we propose a novel state-based approach to insect flight dynamics modeling and automatic insect recognition (AIR). An emerging technology uses noninvasive laser sensors to capture flight information of flying insects in situ. However current computational entomology methods treat these optical signals as static patterns, rather than time series. We propose to evaluate each insect passage as a nonstationary process involving a sequence of pseudo-acoustic frames and model the short-term flight dynamics using a kernel adaptive autoregressive-moving-average (KAARMA) network. Since flight behavior is both nonlinear and nonstationary in nature, dynamic modeling provides a general framework that fully exploits the transitional and contextual information. Using the kernel adaptive ARMA algorithm and spatial clustering, each flight passage is defined to be an ordered sequence of states in a discretized spatiotemporal space. The state transition trajectories are used to construct a finite state machine (FSM) recognizer. The computational efficiency of the automaton classifier eliminates the need to perform numeric computations and enables realtime performance with far-reaching implications in entomology, agriculture, and disease control and prevention. Results show multiclass KAARMA classifier outperforms the state-of-the-art AIR methods, using support vector machine (SVM), deep learning autoencoder, and batch learning, in identifying Zika vector mosquito Aedes aegypti among five species of flying insects, while using significantly more efficient data representation. The extracted FSM from trained KAARMA network retained competitive performance compared to the current computationally expensive implementations.

Dynamic textures (DTs) that represent moving scenes such as flames, smoke, and waves, exhibit fixed dynamics within a period of time and have been successfully modeled using linear dynamic systems (LDS). In this paper, we show that the widely used LDS model can be approximated using a principal component regression (PCR) model with the main advantage of simplicity. Furthermore, to capture the nonlinearity of training frames, we extend traditional PCR to its kernelized version and introduce kernel principal component regression (KPCR) to model and synthesize DTs. To ensure algorithm stability, we remove the standard state model and directly apply the quantized kernel least mean squares algorithm from signal processing domain to approximate the performance achieved with KPCR. We term this improvement kernel adaptive dynamic texture synthesis (KADTS), which also has the benefits of computational and memory efficiency. These advantages make KADTS ideally suited for real-world applications, since the majority of electronic devices, including cell phones and laptops, suffer from limited memory and real-time constraints. We demonstrate, via both theoretical and experimental analyses, the connections between DT synthesis using KPCR and KADTS with a regularization network theory. We also show the superiority of our proposed algorithms for DT synthesis compared with other dynamic system-based benchmarks. MATLAB code is available from our project homepage http://bmal.hust.edu.cn/project/dts.html.

In this paper, we present a novel kernel adaptive recurrent filtering algorithm based on the autoregressive-moving-average (ARMA) model, which is trained with recurrent stochastic gradient descent in the reproducing kernel Hilbert spaces. This kernelized recurrent system, the kernel adaptive ARMA (KAARMA) algorithm, brings together the theories of adaptive signal processing and recurrent neural networks (RNNs), extending the current theory of kernel adaptive filtering (KAF) using the representer theorem to include feedback. Compared with classical feedforward KAF methods, the KAARMA algorithm provides general nonlinear solutions for complex dynamical systems in a state-space representation, with a deferred teacher signal, by propagating forward the hidden states. We demonstrate its capabilities to provide exact solutions with compact structures by solving a set of benchmark nondeterministic polynomial-complete problems involving grammatical inference. Simulation results show that the KAARMA algorithm outperforms equivalent input-space recurrent architectures using first- and second-order RNNs, demonstrating its potential as an effective learning solution for the identification and synthesis of deterministic finite automata.

Since its introduction, kernel adaptive filtering (KAF) has attracted considerable attention in recent years. Its main advantages include universal nonlinear approximation using kernel methods, linearity with convex learning in the Reproducing Kernel Hilbert Space (RKHS), and online adaptation with moderate complexity. Among its applications, the kernel least mean square (KLMS) algorithm deserves particular attention due to its simplicity and effectiveness for learning complex systems. A major drawback of current implementations of KAF is the lack of a simple determination of the certainty of each estimate. In this paper, we present a novel kernel adaptive filtering architecture with confidence intervals. By introducing an auxiliary filter, the variance of each estimate can be computed using stochastic gradient descent in O(N). Results show that the proposed algorithm produces comparable estimates of the mean and the variance functions, using only a fraction of the computation associated with the Gaussian process (GP) prediction, and is more versatile in the cases of time-varying noise variance or heteroskedasticity.