The excessive computational requirements of modern artificial neural networks (ANNs) are posing limitations on the machines that can run them. Sparsification of ANNs is often motivated by time, memory and energy savings only during model inference, yielding no benefits during training. A growing body of work is now focusing on providing the benefits of model sparsification also during training. While these methods greatly improve the training efficiency, the training algorithms yielding the most accurate models still materialize the dense weights, or compute dense gradients during training. We propose an efficient, always-sparse training algorithm with excellent scaling to larger and sparser models, supported by its linear time complexity with respect to the model width during training and inference. Moreover, our guided stochastic exploration algorithm improves over the accuracy of previous sparse training methods. We evaluate our method on the CIFAR-10/100 and ImageNet classification tasks using ResNet, VGG, and ViT models, and compare it against a range of sparsification methods3.

Many computer vision applications rely on interpretable machine learning algorithms to analyze the data collected from various sources. We leverage Hyperdimensional Computing (HDC) as an innovative computational model that mimics key brain functionalities to achieve efficient and robust cognitive learning. We propose HDlm, a novel HDC-based cognitive representation capable of adaptive information association and memorization. HDlm first theoretically expands the HDC mathematics to support selective information association and adaptive memorization. Then, it exploits the proposed operations to support cognitive operations, including set membership, information retrieval, and item comparison. We evaluated our solution for a selection of applications related to visual data representation and sequence matching analysis. Our evaluation shows that HDlm provides more adaptive similarity metrics between objects that lead to better task performance.

HD computing is a symbolic representation system which performs various learning tasks in a highly-parallelizable and binary-centric way by drawing inspiration from concepts in human long-term memory. However, the current HD computing is ineffective in extracting high-level feature information for image data. In this paper, we present a neuro-symbolic approach called NSHD, which integrates CNNs and Hyperdimensional (HD) learning techniques to provide efficient learning with state-of-the-art quality. We devise the HD training procedure, which fully integrates knowledge from the deep learning model through a distillation process with optimized computation costs due to the integration. Our experimental results show that NSHD provides high energy efficiency as compared to CNN, e.g., up to 64% with comparable accuracy, and can outperform the learning quality when more computing resources are allowed. We also show the symbolic nature of the NSHD can make the learning humnan-interpretable by exploiting the property of HD computing.

Recent strides in deep learning have yielded impres-sive practical applications such as autonomous driving, natural language processing, and graph reasoning. However, the sus-ceptibility of deep learning models to subtle input variations, which stems from device imperfections and non-idealities, or adversarial attacks on edge devices, presents a critical challenge. These vulnerabilities hold dual significance-security concerns in critical applications and insights into human-machine sen-sory alignment. Efforts to enhance model robustness encounter resource constraints in the edge and the black box nature of neural networks, hindering their deployment on edge devices. This paper focuses on algorithmic adaptations inspired by the human brain to address these challenges. Hyper Dimensional Computing (HDC), rooted in neural principles, replicates brain functions while enabling efficient, noise-tolerant computation. HDC leverages high-dimensional vectors to encode information, seamlessly blending learning and memory functions. Its trans-parency empowers practitioners, enhancing both robustness and understanding of deployed models. In this paper, we introduce the first comprehensive study that compares the robustness of HDC to white-box malicious attacks to that of deep neural network (DNN) models and the first HDC gradient-based attack in the literature. We develop a framework that enables HDC models to generate gradient-based adversarial examples using state-of-the-art techniques applied to DNNs. Our evaluation shows that our HDC model provides, on average, 19.9% higher robustness than DNNs to adversarial samples and up to 90% robustness improvement against random noise on the weights of the model compared to the DNN.

We provide here a novel method, called hypercolumn sparsification, to achieve high recognition performance for convolutional neural networks (CNNs) despite low-precision weights and activities during both training and test phases. This method is applicable to any CNN architecture that operates on signal patterns (e.g., audio, image, video) to extract information such as class membership. It operates on the stack of feature maps in each of the cascading feature matching and pooling layers through the processing hierarchy of the CNN by an explicit competitive process ( k -WTA, winner take all) that generates a sparse feature vector at each spatial location. This principle is inspired by local brain circuits, where neurons tuned to respond to different patterns in the incoming signals from an upstream region inhibit each other using interneurons, such that only the ones that are maximally activated survive the quenching threshold. We show this process of sparsification is critical for probabilistic learning of low-precision weights and bias terms, thereby making pattern recognition amenable for energy-efficient hardware implementations. Further, we show that hypercolumn sparsification could lead to more data-efficient learning as well as having an emergent property of significantly pruning down the number of connections in the network. A theoretical account and empirical analysis are provided to understand these effects better.

Commonly used brain-inspired models assume that neurons in the network are silent until perturbed by external stimuli. This assumption has created a family of feedforward neural models that progressively detects complex features. However, experimental observations suggest that the brain is spontaneously active where neuronal activity is driven by internal fluctuations in total synaptic input leading to asynchronous and irregular patterns of activity even in the absence of external inputs. The implications of this ground state of the brain is explored for designing brain-inspired circuits that are more driven top-down rather than bottom-up activity leading to a more radical mode of information processing.

During rest, the mammalian cortex displays spontaneous neural activity. Spiking of single neurons during rest has been described as irregular and asynchronous. In contrast, recent in vivo and in vitro population measures of spontaneous activity, using the LFP, EEG, MEG or fMRI suggest that the default state of the cortex is critical, manifested by spontaneous, scale-invariant, cascades of activity known as neuronal avalanches. Criticality keeps a network poised for optimal information processing, but this view seems to be difficult to reconcile with apparently irregular single neuron spiking. Here, we simulate a 10,000 neuron, deterministic, plastic network of spiking neurons. We show that a combination of short- and long-term synaptic plasticity enables these networks to exhibit criticality in the face of intrinsic, i.e. self-sustained, asynchronous spiking. Brief external perturbations lead to adaptive, long-term modification of intrinsic network connectivity through long-term excitatory plasticity, whereas long-term inhibitory plasticity enables rapid self-tuning of the network back to a critical state. The critical state is characterized by a branching parameter oscillating around unity, a critical exponent close to -3/2 and a long tail distribution of a self-similarity parameter between 0.5 and 1.

Modeling of large-scale spiking neural models is an important tool in the quest to understand brain function and subsequently create real-world applications. This paper describes a spiking neural network simulator environment called HRL Spiking Simulator (HRLSim). This simulator is suitable for implementation on a cluster of general purpose graphical processing units (GPGPUs). Novel aspects of HRLSim are described and an analysis of its performance is provided for various configurations of the cluster. With the advent of inexpensive GPGPU cards and compute power, HRLSim offers an affordable and scalable tool for design, real-time simulation, and analysis of large-scale spiking neural networks.

This paper describes a novel architecture for enabling robust and efficient neuromorphic communication. The architecture combines two concepts: 1) synaptic time multiplexing (STM) that trades space for speed of processing to create an intragroup communication approach that is firing rate independent and offers more flexibility in connectivity than cross-bar architectures and 2) a wired multiple input multiple output (MIMO) communication with orthogonal frequency division multiplexing (OFDM) techniques to enable a robust and efficient intergroup communication for neuromorphic systems. The MIMO-OFDM concept for the proposed architecture was analyzed by simulating large-scale spiking neural network architecture. Analysis shows that the neuromorphic system with MIMO-OFDM exhibits robust and efficient communication while operating in real time with a high bit rate. Through combining STM with MIMO-OFDM techniques, the resulting system offers a flexible and scalable connectivity as well as a power and area efficient solution for the implementation of very large-scale spiking neural architectures in hardware.

The thalamus, far from being a simple relay, supports several functions including attention and awareness. Recent spiking models of the thalamus tend to focus on abstract thalamocortical features such as rhythms and synchrony. Here a new spiking retino-thalamic model is presented that reproduces several aspects in visual processing including distinct form and motion processing pathways. Using test and natural image sequences, differences between parvocellular and magnocellular relay neurons are studied. In line with several experimental results, parvocellular neurons are found to be more sensitive to changes in color (necessary for form processing) than temporal frequency (necessary for motion processing) and conversely for magnocellular neurons. This model can in turn be used as input into subsequent cortical models or as a tool to aid in experimentation. Future extensions could include modeling brainstem or cortical influence on thalamic processing, as well as the control of virtual agents.

The challenging aspect of building neuromorphic circuits in mature CMOS technology to match brain-like architectures is two-fold: scalability and connectivity. Scalability means that the circuits have to be expandable to match biological brains in terms of synaptic and neuronal densities. The challenge here is to implement 106 neurons and 1010 synapses with an average fanout of 104, in a square cm of CMOS [1, 2]. Connectivity means that the circuit has to offer the capability to have both short and long range (by physical distance) connections between neurons. A large part of this challenge is how to implement a connectivity of 104 synapses per neuron [3]. Unfortunately, even the exponential transistor density growth being experienced today is not sufficient to realize such massive connectivity and synaptic densities in a traditional CMOS process. Recent approaches to address these challenges have been to integrate CMOS with nanotechnology [4, 5] in order to achieve the required synaptic densities. These solutions use crossbar architectures predominantly but the connectivity challenge still remains a daunting task for such solutions [2, 6]. To meet these challenges, a novel synaptic time-multiplexing (STM) concept was developed along with a neural fabric design [7]. This combination has the advantage of offering greater flexibility and long range connectivity. It also provides a method to overcome the limitations of conventional CMOS technology to match the synaptic density and connectivity requirements found in mammalian brains while maintaining non-linear synapses and learning.

In this letter, a novel critic-like algorithm was developed to extend the synaptic plasticity rule described in Florian (2007) and Izhikevich (2007) in order to solve the problem of learning multiple distal rewards simultaneously. The system is augmented with short-term plasticity (STP) to stabilize the learning dynamics, thereby increasing the system's learning capacity. A theoretical threshold is estimated for the number of distal rewards that this system can learn. The validity of the novel algorithm was verified by computer simulations.

Scalability and connectivity are two key challenges in designing neuromorphic hardware that can match biological levels. In this paper, we describe a neuromorphic system architecture design that addresses an approach to meet these challenges using traditional complementary metal-oxide-semiconductor (CMOS) hardware. A key requirement in realizing such neural architectures in hardware is the ability to automatically configure the hardware to emulate any neural architecture or model. The focus for this paper is to describe the details of such a programmable front-end. This programmable front-end is composed of a neuromorphic compiler and a digital memory, and is designed based on the concept of synaptic time-multiplexing (STM). The neuromorphic compiler automatically translates any given neural architecture to hardware switch states and these states are stored in digital memory to enable desired neural architectures. STM enables our proposed architecture to address scalability and connectivity using traditional CMOS hardware. We describe the details of the proposed design and the programmable front-end, and provide examples to illustrate its capabilities. We also provide perspectives for future extensions and potential applications.

In this paper, we present a spiking neural model that learns spatio-motor transformations. The model is in the form of a multilayered architecture consisting of integrate and fire neurons and synapses that employ spike-timing-dependent plasticity learning rule to enable the learning of such transformations. We developed a simple 2-degree-of-freedom robot-based reaching task which involves the learning of a nonlinear function. Computer simulations demonstrate the capability of such a model for learning the forward and inverse kinematics for such a task and hence to learn spatio-motor transformations. The interesting aspect of the model is its capacity to be tolerant to partial absence of sensory or motor inputs at various stages of learning. We believe that such a model lays the foundation for learning other complex functions and transformations in real-world scenarios.

This paper describes a self-organizing neural model that is capable of controlling the kinematics of robots with redundant degrees of freedom. The self-organized learning process is based on action perception cycles where the robot is perturbed minimally about a given joint configuration and learns to map these perturbations to changes in sensor readings corresponding to these minimal perturbations. This motor babbling phase provides self-generated movement commands that activate correlated sensory, spatial and motor information that are used to learn an internal coordinate transformation between sensory and motor systems. This idea was tested on two different tasks: reaching targets in 2-D space with a three degree of freedom robot and saccading to targets in 3-D with a twelve degree of freedom head-neck-eye system. Computer simulations show that the resulting controller is highly fault-tolerant and robust to previously unseen disturbances much like biological systems.

We present a fuzzy clustering approach to collaborative filtering. Our approach allows for users to be clustered into multiple user groups. Furthermore, our approach is active in that it can rapidly adapt to both short and long term user interest changes. Our approach is capable of on-line collaborative filtering with simultaneous clustering at the document content level, user group level, as well as document clustering based on similarity of user interests. We demonstrate the various features of the approach using a synthetic example.

Object fingerprinting and identification is a critical part of effective visual surveillance systems. In this paper, we present an approach to actively learn the object models in order to fingerprint the objects. Our approach uses a view-based classifier cascade that actively learns to recognize the generic class of the object. Salient features unique to the specific instance of the selected class of objects are modeled using fuzzy attribute relational graphs. These graphs are also adapted to represent object information gathered from multiple views. Preliminary results are quite promising and extensive studies are underway to ascertain the use of the system in more complicated scenarios.

The paper proposes a new set of fuzzy features based on symmetry of edges for improving the accuracy of detecting intruders. We show that the proposed fuzzy edge-symmetry feature-based classifier is comparable to the detection accuracy of a multi-scale wavelet feature system for intruder detection. We also present two approaches to fusing the results of classifiers trained independently on the edge-symmetry and wavelet features. Experimental results clearly indicate the improvement in system performance when the results of the two classifiers are fused.

In this paper we present an approach that is capable of online learning and automatic generation of a fuzzy expert system for high dimensional classification problems. The novel part of our system is a new online learning rule. Unlike other learning systems, this learning rule makes our system scale robustly with input space dimensions and thus suitable for high dimensional data. The algorithm is also able to extract knowledge in an online fashion in the form of fuzzy rules that are comprehensible, compact, and accurate.

In this paper, we describe the design of an artificial neural network for spatiotemporal pattern recognition and recall. This network has a five-layered architecture and operates in two modes: pattern learning and recognition mode, and pattern recall mode. In pattern learning and recognition mode, the network extracts a set of topologically and temporally correlated features from each spatiotemporal input pattern based on a variation of Kohonen's self-organizing maps. These features are then used to classify the input into categories based on the fuzzy ART network. In the pattern recall mode, the network can reconstruct any of the learned categories when the appropriate category node is excited or probed. The network performance was evaluated via computer simulations of time-varying, two-dimensional and three-dimensional data. The results show that the network is capable of both recognition and recall of spatiotemporal data in an on-line and self-organized fashion. The network can also classify repeated events in the spatiotemporal input and is robust to noise in the input such as distortions in the spatial and temporal content.

We propose a reconfigurable gripper that consists of two parallel plates whose distance can be adjusted by a computer-controlled actuator. The bottom plate is a bare plane, and the top plate carries a rectangular grid of actuated pins that can translate in discrete increments under computer control. We propose to use this gripper to immobilize objects through frictionless contacts with three of the pins and the bottom plate. We present an efficient grasp planning algorithm, describe the design of the gripper, which is currently under construction, and report preliminary simulation experiments.

We propose a novel neural network, called the self-organized invertible map (SOIM), that is capable of learning many-to-one functionals mappings in a self-organized and online fashion. The design and performance of the SOIM are highlighted by learning a many-to-one functional mapping that exists in active vision for spatial representation of three-dimensional point targets. The learned spatial representation is invariant to changing camera configurations. The SOIM also possesses an invertible property that can be exploited for active vision. An efficient and experimentally feasible method was devised for learning this representation on a real active vision system. The proof of convergence during learning as well as conditions for invariance of the learned spatial representation are derived and then experimentally verified using the active vision system. We also demonstrate various active vision applications that benefit from the properties of the mapping learned by SOIM.

Robots that use an active camera system for visual feedback can achieve greater flexibility, including the ability to operate in a dynamically changing environment. Incorporating active vision into a robot control loop involves some inherent difficulties, including calibration, and the need for redefining the goal as the camera configuration changes. In this paper, we propose a novel self-organizing neural network (SOIM) that learns a calibration-free spatial representation of 3D point targets in a manner that is invariant to changing camera configurations. This representation is used to develop a new framework for robot control with active vision. The salient feature of this framework is that it decouples active camera control from robot control. The feasibility of this approach is explored with the help of computer simulations and experiments with the University of Illinois Active Vision System (UIAVS).

A novel invariant pattern recognition machine is proposed based on a modified ART architecture. Invariance is achieved by adding a new layer called F/sub 3/, beyond the F/sub 2/ layer in the ART architecture. The design of the weight connections between the nodes of the F/sub 2/ layer and the cells of the F/sub 3/ layer are similar to the invariance net. Computer simulations show that the model is not only invariant to translations and rotations of 2-D binary images but also noise-tolerant to these transformed images.>