Event cameras form a fundamental foundation for visual perception in scenes characterized by high speed and a wide dynamic range. Although deep learning techniques have achieved remarkable success in estimating event-based optical flow, existing methods have not adequately addressed the significance of temporal information in capturing spatiotemporal features. Due to the dynamics of spiking neurons in SNNs, which preserve important information while forgetting redundant information over time, they are expected to outperform analog neural networks (ANNs) with the same architecture and size in sequential regression tasks. In addition, SNNs on neuromorphic hardware achieve advantages of extremely low power consumption. However, present SNN architectures encounter issues related to limited generalization and robustness during training, particularly in noisy scenes. To tackle these problems, this study introduces an innovative spike-based self-supervised learning algorithm known as SeLHIB, which leverages the information bottleneck theory. By utilizing event-based camera inputs, SeLHIB enables robust estimation of optical flow in the presence of noise. To the best of our knowledge, this is the first proposal of a self-supervised information bottleneck learning strategy based on SNNs. Furthermore, we develop spike-based self-supervised algorithms with nonlinear and high-order information bottleneck learning that employs nonlinear and high-order mutual information to enhance the extraction of relevant information and eliminate redundancy. We demonstrate that SeLHIB significantly enhances the generalization ability and robustness of optical flow estimation in various noise conditions. In terms of energy efficiency, SeLHIB achieves 90.44% and 45.70% cut down of energy consumption compared to its counterpart ANN and counterpart SNN models, while attaining 33.78% lower AEE (MVSEC), 5.96% lower RSAT (ECD) and 6.21% lower RSAT (HQF) compared to the counterpart ANN implementations with the same sizes and architectures.

Open-vocabulary video instance segmentation strives to segment and track instances belonging to an open set of categories in a videos. The vision-language model Contrastive Language-Image Pre-training (CLIP) has shown robust zero-shot classification ability in image-level open-vocabulary tasks. In this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts frozen CLIP and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification. Given a set of initial queries, class-agnostic mask generation introduces a pixel decoder and a transformer decoder on CLIP pre-trained image encoder to predict query masks and corresponding object scores and mask IoU scores. Then, temporal topK-enhanced matching performs query matching across frames using the K mostly matched frames. Finally, weighted open-vocabulary classification first employs mask pooling to generate query visual features from CLIP pre-trained image encoder, and second performs weighted classification using object scores and mask IoU scores. Our CLIP-VIS does not require the annotations of instance categories and identities. The experiments are performed on various video instance segmentation datasets, which demonstrate the effectiveness of our proposed method, especially for novel categories. When using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of 32.2% and 40.2% on the validation set of LV-VIS dataset, which outperforms OV2Seg by 11.1% and 23.9% respectively. We will release the source code and models athttps://github.com/zwq456/CLIP-VIS.git.

Brain-inspired computing technique presents a promising approach to prompt the rapid development of artificial general intelligence (AGI). As one of the most critical aspects, spiking neural networks (SNNs) have demonstrated superiority for AGI, such as low power consumption. Effective training of SNNs with high generalization ability, high robustness, and low power consumption simultaneously is a significantly challenging problem for the development and success of applications of spike-based machine intelligence. In this research, we present a novel and flexible learning framework termed high-order spike-based information bottleneck (HOSIB) leveraging the surrogate gradient technique. The presented HOSIB framework, including second-order and third-order formation, i.e., second-order information bottleneck (SOIB) and third-order information bottleneck (TOIB), comprehensively explores the common latent architecture and the spike-based intrinsic information and discards the superfluous information in the data, which improves the generalization capability and robustness of SNN models. Specifically, HOSIB relies on the information bottleneck (IB) principle to prompt the sparse spike-based information representation and flexibly balance its exploitation and loss. Extensive classification experiments are conducted to empirically show the promising generalization ability of HOSIB. Furthermore, we apply the SOIB and TOIB algorithms in deep spiking convolutional networks to demonstrate their improvement in robustness with various categories of noise. The experimental results prove the HOSIB framework, especially TOIB, can achieve better generalization ability, robustness and power efficiency in comparison with the current representative studies.

The brain possesses the remarkable ability to seamlessly integrate perception with decision making within a dynamically changing environment in a fault-tolerant, end-to-end manner. This extraordinary capability offers a compelling solution for brain-inspired intelligence, replete with the advantages of end-to-end decision making: robustness, high accuracy, real-time responsiveness, autonomous intelligence, and a high degree of biological plausibility. Neuromorphic computing stands as a promising avenue for brain-inspired intelligence through the harmonious co-design of algorithms and hardware, aimed at unlocking full potential. This article introduces a comprehensive neuromorphic computing framework for end-to-end intelligence. It introduces the quadruplet spike-timing-dependent plasticity, which serves as a cornerstone for perceptual to decision-making tasks. A fault-tolerant neuromorphic routing strategy is presented to fortify the framework’s robustness. Empirical results underscore its impressive attributes with high accuracy, robustness, fault tolerance, and minimal computational latency when orchestrating end-to-end decision-making alongside visual perception. This study marks a pioneering effort in unified, fault-tolerant neuromorphic framework engineered for brain-inspired end-to-end intelligent tasks, merging visual perception with adaptive decision making. Such an endeavor is profoundly meaningful, as it propels the development of artificial general intelligence, holding vast implications for the field’s advancement.

Although deep learning-based change detection (CD) methods achieve great success in remote sensing images, they still suffer from two main challenges. First, popular Convolutional Neural Networks (CNNs) are weak in extracting discriminated features focusing on changed regions, since most methods ignore the multi-frequency components of bi-temporal images. Second, although existing CD methods employ the Transformer structure to capture long-range dependency for global feature representation, it is difficult for them to simultaneously take into account the long-range dependency of changed objects at various scales. To address the above issues, we propose a triple change detection network (TCD-Net) via joint multi-frequency and full-scale Swin-Transformer. The proposed TCD-Net has two main advantages. First, we propose a multi-frequency channel attention (MFCA) module to boost the ability of modeling the channel correlation, which can compensate for the problem of insufficient feature representation caused by only performing global average pooling (GAP). Furthermore, a joint multi-frequency difference feature enhancement (JM-DFE) guiding block is proposed to improve the boundary quality and the position awareness of truly changed objects, which can effectively extract channel features of multi-frequency information and thus improve the discriminative ability of features. Second, unlike Siamese-based structures, we propose a full-scale Swin-Transformer (FST) module as the third branch to model and aggregate the long-range dependency of multi-scale changed objects, which can alleviate the missed detections of small objects and achieve more compact changed regions effectively. Experiments on three public CD datasets exhibit that the proposed TCD-Net achieves better CD accuracy with smaller model complexity than state-of-the-art methods. The code is publicly available at https://github.com/RSCD-mz/TCD-Net.

To investigate the diversified technologies in Internet of Vehicles (IoVs) under intelligent edge computing, brain-inspired computing techniques are proposed in this study, which is a promising biologically inspired method by using brain cognition mechanism for various applications. A neuromorphic approach in a scalable and fault-tolerant framework is presented, targeting to realize the navigation function for the edge computing in IoV applications. A novel fault-tolerant address event representation approach is proposed for the spike information routing, which makes the presented model both scalable and fault-tolerant. Experimental results reveal that the proposed approaches can enhance the communication distance, the load balancing and the maximum throughput of the neuromorphic system accordingly. Based on the proposed neuromorphic model, the effects of the dopamine level are investigated. Besides, the results show that the proposed work can realize the accurate obstacle avoidance for the edge IoV computing, and the performance of the proposed network is superior to the network without the proposed scalable and fault-tolerant design. Therefore, the proposed IoV model provides an experimental basis for the improvement of the IoV system.

Spiking neural networks (SNNs) have garnered increased attention in the field of artificial general intelligence (AGI) research due to their low power consumption, high computational efficiency, and low latency induced by their event-driven and sparse communication features. However, efficiently and robustly training an SNN presents a challenge. In this study, we introduce a novel framework for spike-based machine learning called spike-based nonlinear information bottleneck (SNIB). This framework utilizes an information-theoretic learning (ITL) approach and a surrogate gradient learning (SGL) method to achieve robust, accurate, and low-power performance. The proposed SNIB framework includes three variants: 1) squared information bottleneck (SIB); 2) cubic information bottleneck (CIB); and 3) quartic information bottleneck (QIB) strategies, which use a mapping mechanism to compress spiking representations. We systematically evaluate these strategies using different types of input noise and neuromorphic hardware noise. Our experimental results demonstrate that all three strategies effectively enhance the robustness of SGL in SNN architectures. Furthermore, SNIB can significantly reduce the power consumption of SNNs. As a result, SNIB offers a new and significant perspective for hardware-constrained general mobile devices for embedded edge intelligence and represents a progressive step toward realizing AGI.

The further exploration of the neural mechanisms underlying the biological activities of the human brain depends on the development of large-scale spiking neural networks (SNNs) with different categories at different levels, as well as the corresponding computing platforms. Neuromorphic engineering provides approaches to high-performance biologically plausible computational paradigms inspired by neural systems. In this article, we present a biological-inspired cognitive supercomputing system (BiCoSS) that integrates multiple granules (GRs) of SNNs to realize a hybrid compatible neuromorphic platform. A scalable hierarchical heterogeneous multicore architecture is presented, and a synergistic routing scheme for hybrid neural information is proposed. The BiCoSS system can accommodate different levels of GRs and biological plausibility of SNN models in an efficient and scalable manner. Over four million neurons can be realized on BiCoSS with a power efficiency of 2.8k larger than the GPU platform, and the average latency of BiCoSS is 3.62 and 2.49 times higher than conventional architectures of digital neuromorphic systems. For the verification, BiCoSS is used to replicate various biological cognitive activities, including motor learning, action selection, context-dependent learning, and movement disorders. Comprehensively considering the programmability, biological plausibility, learning capability, computational power, and scalability, BiCoSS is shown to outperform the alternative state-of-the-art works for large-scale SNN, while its real-time computational capability enables a wide range of potential applications.

Neuromorphic computing is a promising technology that realizes computation based on event-based spiking neural networks (SNNs). However, fault-tolerant on-chip learning remains a challenge in neuromorphic systems. This study presents the first scalable neuromorphic fault-tolerant context-dependent learning (FCL) hardware framework. We show how this system can learn associations between stimulation and response in two context-dependent learning tasks from experimental neuroscience, despite possible faults in the hardware nodes. Furthermore, we demonstrate how our novel fault-tolerant neuromorphic spike routing scheme can avoid multiple fault nodes successfully and can enhance the maximum throughput of the neuromorphic network by 0.9%-16.1% in comparison with previous studies. By utilizing the real-time computational capabilities and multiple-fault-tolerant property of the proposed system, the neuronal mechanisms underlying the spiking activities of neuromorphic networks can be readily explored. In addition, the proposed system can be applied in real-time learning and decision-making applications, brain-machine integration, and the investigation of brain cognition during learning.

The cerebellum plays a vital role in motor learning and control with supervised learning capability, while neuromorphic engineering devises diverse approaches to high-performance computation inspired by biological neural systems. This article presents a large-scale cerebellar network model for supervised learning, as well as a cerebellum-inspired neuromorphic architecture to map the cerebellar anatomical structure into the large-scale model. Our multinucleus model and its underpinning architecture contain approximately 3.5 million neurons, upscaling state-of-the-art neuromorphic designs by over 34 times. Besides, the proposed model and architecture incorporate 3411k granule cells, introducing a 284 times increase compared to a previous study including only 12k cells. This large scaling induces more biologically plausible cerebellar divergence/convergence ratios, which results in better mimicking biology. In order to verify the functionality of our proposed model and demonstrate its strong biomimicry, a reconfigurable neuromorphic system is used, on which our developed architecture is realized to replicate cerebellar dynamics during the optokinetic response. In addition, our neuromorphic architecture is used to analyze the dynamical synchronization within the Purkinje cells, revealing the effects of firing rates of mossy fibers on the resonance dynamics of Purkinje cells. Our experiments show that real-time operation can be realized, with a system throughput of up to 4.70 times larger than previous works with high synaptic event rate. These results suggest that the proposed work provides both a theoretical basis and a neuromorphic engineering perspective for brain-inspired computing and the further exploration of cerebellar learning.

Multicompartment emulation is an essential step to enhance the biological realism of neuromorphic systems and to further understand the computational power of neurons. In this paper, we present a hardware efficient, scalable, and real-time computing strategy for the implementation of large-scale biologically meaningful neural networks with one million multi-compartment neurons (CMNs). The hardware platform uses four Altera Stratix III field-programmable gate arrays, and both the cellular and the network levels are considered, which provides an efficient implementation of a large-scale spiking neural network with biophysically plausible dynamics. At the cellular level, a cost-efficient multi-CMN model is presented, which can reproduce the detailed neuronal dynamics with representative neuronal morphology. A set of efficient neuromorphic techniques for single-CMN implementation are presented with all the hardware cost of memory and multiplier resources removed and with hardware performance of computational speed enhanced by 56.59% in comparison with the classical digital implementation method. At the network level, a scalable network-on-chip (NoC) architecture is proposed with a novel routing algorithm to enhance the NoC performance including throughput and computational latency, leading to higher computational efficiency and capability in comparison with state-of-the-art projects. The experimental results demonstrate that the proposed work can provide an efficient model and architecture for large-scale biologically meaningful networks, while the hardware synthesis results demonstrate low area utilization and high computational speed that supports the scalability of the approach.

The investigation of the human intelligence, cognitive systems and functional complexity of human brain is significantly facilitated by high-performance computational platforms. In this paper, we present a real-time digital neuromorphic system for the simulation of large-scale conductance-based spiking neural networks (LaCSNN), which has the advantages of both high biological realism and large network scale. Using this system, a detailed large-scale cortico-basal ganglia-thalamocortical loop is simulated using a scalable 3-D network-on-chip (NoC) topology with six Altera Stratix III field-programmable gate arrays simulate 1 million neurons. Novel router architecture is presented to deal with the communication of multiple data flows in the multinuclei neural network, which has not been solved in previous NoC studies. At the single neuron level, cost-efficient conductance-based neuron models are proposed, resulting in the average utilization of 95% less memory resources and 100% less DSP resources for multiplier-less realization, which is the foundation of the large-scale realization. An analysis of the modified models is conducted, including investigation of bifurcation behaviors and ionic dynamics, demonstrating the required range of dynamics with a more reduced resource cost. The proposed LaCSNN system is shown to outperform the alternative state-of-the-art approaches previously used to implement the large-scale spiking neural network, and enables a broad range of potential applications due to its real-time computational power.

One of the challenging problems in real-time control of movement disorders is the effective handling of time-variant brain activities that involve stochastic functional networks with nonlinear dynamics. For such challenges in neuromodulation tasks, fuzzy logic control (FLC) has shown significant potential. The objective of this paper is to present a FLC-based strategy to treat pathological symptoms of movement-disorder with higher performance. The strategy is two-fold: first, develop a design methodology for the FLC system that can robustly control pathological conditions and significantly improve control performance; and second, develop a hardware-efficient implementation for real-time neuromodulation applications. To enhance control performance, a hidden variable in the neural network that can be estimated using an unscented Kalman filter is identified as a feedback variable. In comparison with state-of-the-art schemes, the proposed design can adaptively optimize the control signals without requiring particular information of the controlled plant, thus avoiding repeated determinations of controller parameters. A field-programmable gate array is used for the reconfigurable realization of the entire control strategy based on a modification of the original neural network. The presented design, with enhanced control performance and higher hardware efficiency, has significant potential for clinical treatment of movement disorders and offers a new perspective on applications in the fields of neural control engineering and brain-machine interfaces.