Backpropagation (BP), the standard learning algorithm for artificial neural networks, is often considered biologically implausible. In contrast, the standard learning algorithm for predictive coding (PC) models in neuroscience, known as the inference learning algorithm (IL), is a promising, bio-plausible alternative. However, several challenges and questions hinder IL's application to real-world problems. For example, IL is computationally demanding, and without memory-intensive optimizers like Adam, IL may converge to poor local minima. Moreover, although IL can reduce loss more quickly than BP, the reasons for these speedups or their robustness remains unclear. In this paper, we tackle these challenges by 1) altering the standard implementation of PC circuits to substantially reduce computation, 2) developing a novel optimizer that improves the convergence of IL without increasing memory usage, and 3) establishing theoretical results that help elucidate the conditions under which IL is sensitive to second and higher-order information.

Spiking wavefront planners for navigation demonstrate biologically plausible behavior when exploring and planning paths through an environment. Not present in these models, however, is the replay of previous experiences observed in hippocampal sharp wave ripple complexes (SWRs) during sleep and wake resting states. This work implements a memory replay algorithm in a spiking wavefront model, and investigates different theories of replay selection. Results indicate that the addition of replay in the spiking wavefront model improves the speed at which the agent learns the environment, and the ability to adapt to change. Furthermore, selection of replays based on its effectiveness in updating model weights leads to greater improvement when compared to a uniformly weighted selection.

Axonal plasticity describes the biological phenomenon in which the myelin sheath thickness and the amplification of a signal change due to experience. Recent studies show this to be important for sequence learning and synchronization of temporal information. In spiking neural networks (SNNs), the time a spike travels from the presynaptic neuron along the axon until it reaches a postsynaptic neuron is an essential principle of how SNNs encode information. In simulators for large scale SNN models such as CARLsim, this time is modeled as synaptic delays with discrete values from one to several milliseconds. To simulate neural activity in large-scale SNNs efficiently, delays are transformed as indices to optimized structures that are built once before the simulation starts. As a consequence, and in contrast to synaptic weights, delays are not directly accessible as scalar data in the runtime memory. In the present paper, we introduce axonal delay learning rules in the SNN simulator CARLsim that can be updated during runtime. To demonstrate this feature, we implement the recent E-Prop learning rule in a recurrent SNN capable of flexible navigation. Compared to other studies for axonal plasticity that are based on LIF neurons, we also develop the SNN based on the more biologically realistic Izhikevich neural model. The present work serves as reference implementation for neuromorphic hardware that encode delays and serves as an interesting alternative to synaptic plasticity.

Mature simulation systems for Spiking Neural Networks (SNNs) become more relevant than ever for understanding the brain and supporting neuromorphic computing. The CARL-sim SNN platform is one of the first Open Source simulation systems that utilized CUDA GPUs to address the tremendous parallel processing demands of natural brains. It has evolved over almost a decade in numerous scientific research projects requiring efficient biologically plausible modeling at scale. With its sixth major release, CARLsim 6 respects this legacy by supporting the latest versions of operating systems, development tool chains, multi-core computers, and of course GPUs. It runs on a range of platforms; from Notebooks up to the NVIDIA DGX-A100 supercomputer, and is used in biologically plausible simulations of the hippocampus and neocortex. The latest version has added flexibility for incorporating long-term and short-term synaptic plasticity. Neuromodulation is an important property of neurobiology that can lead to rapid few shot learning, network rewiring, and neural activity modulation. Because of this, CARLsim 6 now supports four multiple neuromodulators for simulating neural excitability and synaptic plasticity.

Backpropagation (BP) is the most successful and widely used algorithm in deep learning. However, the computations required by BP are challenging to reconcile with known neurobiology. This difficulty has stimulated interest in more biologically plausible alternatives to BP. One such algorithm is the inference learning algorithm (IL). IL trains predictive coding models of neural circuits and has achieved equal performance to BP on supervised and auto-associative tasks. In contrast to BP, however, the mathematical foundations of IL are not well-understood. Here, we develop a novel theoretical framework for IL. Our main result is that IL closely approximates an optimization method known as implicit stochastic gradient descent (implicit SGD), which is distinct from the explicit SGD implemented by BP. Our results further show how the standard implementation of IL can be altered to better approximate implicit SGD. Our novel implementation considerably improves the stability of IL across learning rates, which is consistent with our theory, as a key property of implicit SGD is its stability. We provide extensive simulation results that further support our theoretical interpretations and find IL achieves quicker convergence when trained with mini-batch size one while performing competitively with BP for larger mini-batches when combined with Adam.

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

Neuromorphic computing systems are embracing memristors to implement high density and low power synaptic storage as crossbar arrays in hardware. These systems are energy efficient in executing Spiking Neural Networks (SNNs). We observe that long bitlines and wordlines in a memristive crossbar are a major source of parasitic voltage drops, which create current asymmetry. Through circuit simulations, we show the significant endurance variation that results from this asymmetry. Therefore, if the critical memristors (ones with lower endurance) are overutilized, they may lead to a reduction of the crossbar's lifetime. We propose eSpine, a novel technique to improve lifetime by incorporating the endurance variation within each crossbar in mapping machine learning workloads, ensuring that synapses with higher activation are always implemented on memristors with higher endurance, and vice versa. eSpine works in two steps. First, it uses the Kernighan-Lin Graph Partitioning algorithm to partition a workload into clusters of neurons and synapses, where each cluster can fit in a crossbar. Second, it uses an instance of Particle Swarm Optimization (PSO) to map clusters to tiles, where the placement of synapses of a cluster to memristors of a crossbar is performed by analyzing their activation within the workload. We evaluate eSpine for a state-of-the-art neuromorphic hardware model with phase-change memory (PCM)-based memristors. Using 10 SNN workloads, we demonstrate a significant improvement in the effective lifetime.

Despite the recent success of deep reinforcement learning (RL), domain adaptation remains an open problem. Although the generalization ability of RL agents is critical for the real-world applicability of Deep RL, zero-shot policy transfer is still a challenging problem since even minor visual changes could make the trained agent completely fail in the new task. To address this issue, we propose a two-stage RL agent that first learns a latent unified state representation (LUSR) which is consistent across multiple domains in the first stage, and then do RL training in one source domain based on LUSR in the second stage. The cross-domain consistency of LUSR allows the policy acquired from the source domain to generalize to other target domains without extra training. We first demonstrate our approach in variants of CarRacing games with customized manipulations, and then verify it in CARLA, an autonomous driving simulator with more complex and realistic visual observations. Our results show that this approach can achieve state-of-the-art domain adaptation performance in related RL tasks and outperforms prior approaches based on latent-representation based RL and image-to-image translation.

In rodent navigational studies, spatial responses have been identified in both the hippocampal subregion CA1 and the subiculum (SUB), but these two brain regions appear to encode spatial features differently. Place fields of SUB place cells are larger and less specific than CA1. Additionally, SUB neurons exhibit stronger directional modulation for heading and axes of travel. Based on neural and behavioral data recorded as rats perform a navigational task on a “triple-T” maze, we present a spiking neural network modeling framework to replicate response properties observed in the CA1 and SUB. The parameters of Spike Timing Dependent Plasticity and homeostatic scaling (STDP-H) were evolved such that the response of the two different SNNs resembled recordings from CA1 and SUB when rats traversed the triple-T maze. Our results suggest that positional input may be more influential in forming CA1 place cells, while the SUB appears to integrate both allocentric positional information and self-motion cues to encode “kinds of places”. Furthermore, our results predict that the different spatial responses in these regions may be due in part to different STDP-H learning parameters. The framework presented here could be used as an automated parameter tuning system for replicating responses in other brain regions.

Neuromorphic computing systems execute machine learning tasks designed with spiking neural networks. These systems are embracing non-volatile memory to implement high-density and low-energy synaptic storage. Elevated voltages and currents needed to operate non-volatile memories cause aging of CMOS-based transistors in each neuron and synapse circuit in the hardware, drifting the transistor’s parameters from their nominal values. If these circuits are used continuously for too long, the parameter drifts cannot be reversed, resulting in permanent degradation of circuit performance over time, eventually leading to hardware faults. Aggressive device scaling increases power density and temperature, which further accelerates the aging, challenging the reliable operation of neuromorphic systems. Existing reliability-oriented techniques periodically de-stress all neuron and synapse circuits in the hardware at fixed intervals, assuming worst-case operating conditions, without actually tracking their aging at run-time. To de-stress these circuits, normal operation must be interrupted, which introduces latency in spike generation and propagation, impacting the inter-spike interval and hence, performance (e.g., accuracy). We observe that in contrast to long-term aging, which permanently damages the hardware, short-term aging in scaled CMOS transistors is mostly due to bias temperature instability. The latter is heavily workload-dependent and, more importantly, partially reversible. We propose a new architectural technique to mitigate the aging-related reliability problems in neuromorphic systems by designing an intelligent run-time manager (NCRTM), which dynamically de-stresses neuron and synapse circuits in response to the short-term aging in their CMOS transistors during the execution of machine learning workloads, with the objective of meeting a reliability target. NCRTM de-stresses these circuits only when it is absolutely necessary to do so, otherwise reducing the performance impact by scheduling de-stress operations off the critical path. We evaluate NCRTM with state-of-the-art machine learning workloads on a neuromorphic hardware. Our results demonstrate that NCRTM significantly improves the reliability of neuromorphic hardware, with marginal impact on performance.

Disentangling the sources of visual motion in a dynamic scene during self-movement or ego motion is important for autonomous navigation and tracking. In the dynamic image segments of a video frame containing independently moving objects, optic flow relative to the next frame is the sum of the motion fields generated due to camera and object motion. The traditional ego-motion estimation methods assume the scene to be static, and the recent deep learning-based methods do not separate pixel velocities into object- and ego-motion components. We propose a learning-based approach to predict both ego-motion parameters and object-motion field (OMF) from image sequences using a convolutional autoencoder while being robust to variations due to the unconstrained scene depth. This is achieved by: 1) training with continuous ego-motion constraints that allow solving for ego-motion parameters independently of depth and 2) learning a sparsely activated overcomplete ego-motion field (EMF) basis set, which eliminates the irrelevant components in both static and dynamic segments for the task of ego-motion estimation. In order to learn the EMF basis set, we propose a new differentiable sparsity penalty function that approximates the number of nonzero activations in the bottleneck layer of the autoencoder and enforces sparsity more effectively than L1- and L2-norm-based penalties. Unlike the existing direct ego-motion estimation methods, the predicted global EMF can be used to extract OMF directly by comparing it against the optic flow. Compared with the state-of-the-art baselines, the proposed model performs favorably on pixelwise object- and ego-motion estimation tasks when evaluated on real and synthetic data sets of dynamic scenes.

We present PyCARL, a PyNN-based common Python programming interface for hardware-software cosimulation of spiking neural network (SNN). Through PyCARL, we make the following two key contributions. First, we provide an interface of PyNN to CARLsim, a computationally- efficient, GPU-accelerated and biophysically-detailed SNN simulator. PyCARL facilitates joint development of machine learning models and code sharing between CARLsim and PyNN users, promoting an integrated and larger neuromorphic community. Second, we integrate cycle-accurate models of state-of-the-art neuromorphic hardware such as TrueNorth, Loihi, and DynapSE in PyCARL, to accurately model hardware latencies, which delay spikes between communicating neurons, degrading performance of machine learning models. PyCARL allows users to analyze and optimize the performance difference between software-based simulation and hardware-oriented simulation. We show that system designers can also use PyCARL to perform design-space exploration early in the product development stage, facilitating faster time-to-market of neuromorphic products.

A robot operating in multiple settings must develop stable as well as flexible representations of the tasks and contexts associated with their environments. Taking inspiration from neurobiology, we apply a neural network model of schemas and memory consolidation to train the Toyota Human Support Robot to find and retrieve objects in indoor settings. We define schemas to be collections of objects bound together by a common context. In this case, the robot must learn schemas associated with rooms found in a school based on objects typically found in those rooms. Because the model develops schema representations for each room, the robot can rapidly perform object retrieval tasks associated with familiar schemas and disambiguate the tasks by context. Our experiment explores the effects of the model in an embodied setting and shows the benefits of applying research in memory consolidation to contextual awareness in robotics.

Robots and self-driving vehicles face a number of challenges when navigating through real environments. Successful navigation in dynamic environments requires prioritizing subtasks and monitoring resources. Animals are under similar constraints. It has been shown that the neuromodulator serotonin (5-HT) regulates impulsiveness and patience in animals. In the present paper, we take inspiration from the serotonergic system and apply it to the task of robot navigation. In a set of outdoor experiments, we show how changing the level of patience can affect the amount of time the robot will spend searching for a desired location. To navigate GPS compromised environments, we introduce a deep reinforcement learning paradigm in which the robot learns to follow sidewalks. This may further regulate a tradeoff between a smooth long route and a rough shorter route. Using patience as a parameter may be beneficial for autonomous systems under time pressure.

Terrain classification is important for outdoor path planning, mapping, and navigation. We developed a reservoir-based spiking neural network (r-SNN) to classify three terrain types (i.e. grass, dirt, and road) in a botanical garden. It included a recurrent layer and a supervised layer. The input spike trains to the recurrent layer were generated from linear accelerometer and gyroscope sensor signals as well as camera frames from an Android smartphone that controlled a ground robot. Compared to a Support Vector Machine (SVM) model and a 3-layer (3L) logistic regression model, our r-SNN method generated better prediction accuracy without reliance on a time window of data. Using both images and sensors as input, the test accuracy of the r-SNN was over 95%, which was significantly better than the SVM and the 3L logistic regression. Because the r-SNN is compatible with neuromorphic hardware, our proposed method could be part of a biologically-inspired power-efficient autonomous robot navigation system.

In this article, we explore neurobiological principles that could be deployed in systems requiring self-preservation, adaptive control, and contextual awareness. We start with low-level control for sensor processing and motor reflexes. We then discuss how critical it is at an intermediate level to maintain homeostasis and predict system set points. We end with a discussion at a high-level, or cognitive level, where planning and prediction can further monitor the system and optimize performance. We emphasize the information flow between these levels both from a systems neuroscience and an engineering point of view. Throughout the paper, we describe the brain systems that carry out these functions and provide examples from artificial intelligence, machine learning, and robotics that include these features. Our goal is to show how biological organisms performing self-monitoring can inspire the design of autonomous and embedded systems.

Neuromorphic hardware implements biological neurons and synapses to execute a spiking neural network (SNN)-based machine learning. We present SpiNeMap, a design methodology to map SNNs to crossbar-based neuromorphic hardware, minimizing spike latency and energy consumption. SpiNeMap operates in two steps: SpiNeCluster and SpiNePlacer. SpiNeCluster is a heuristic-based clustering technique to partition an SNN into clusters of synapses, where intracluster local synapses are mapped within crossbars of the hardware and intercluster global synapses are mapped to the shared interconnect. SpiNeCluster minimizes the number of spikes on global synapses, which reduces spike congestion and improves application performance. SpiNePlacer then finds the best placement of local and global synapses on the hardware using a metaheuristic-based approach to minimize energy consumption and spike latency. We evaluate SpiNeMap using synthetic and realistic SNNs on a state-of-the-art neuromorphic hardware. We show that SpiNeMap reduces average energy consumption by 45% and spike latency by 21%, compared to the best-performing SNN mapping technique.

Supported by recent computational studies, there is increasing evidence that a wide range of neuronal responses can be understood as an emergent property of nonnegative sparse coding (NSC), an efficient population coding scheme based on dimensionality reduction and sparsity constraints.We review evidence that NSC might be employed by sensory areas to efficiently encode external stimulus spaces, by some associative areas to conjunctively represent multiple behaviorally relevant variables, and possibly by the basal ganglia to coordinate movement.In addition, NSC might provide a useful theoretical framework under which to understand the often complex and nonintuitive response properties of neurons in other brain areas.Although NSC might not apply to all brain areas (for example, motor or executive function areas) the success of NSC-based models, especially in sensory areas, warrants further investigation for neural correlates in other regions. Author summaryBrains face the fundamental challenge of extracting relevant information from high-dimensional external stimuli in order to form the neural basis that can guide an organism's behavior and its interaction with the world.One potential approach to addressing this challenge is to reduce the number of variables required to represent a particular input space (i.e., dimensionality reduction).We review compelling evidence that a range of neuronal responses can be understood as an emergent property of nonnegative sparse coding (NSC)-a form of efficient population coding due to dimensionality reduction and sparsity constraints.

Large-scale spiking neural network (SNN) simulations are challenging to implement, due to the memory and computation required to iteratively process the large set of neural state dynamics and updates. To meet these challenges, we have developed CARLsim 4, a user-friendly SNN library written in C++ that can simulate large biologically detailed neural networks. Improving on the efficiency and scalability of earlier releases, the present release allows for the simulation using multiple GPUs and multiple CPU cores concurrently in a heterogeneous computing cluster. Benchmarking results demonstrate simulation of 8.6 million neurons and 0.48 billion synapses using 4 GPUs and up to 60x speedup for multi-GPU implementations over a single-threaded CPU implementation, making CARLsim 4 well-suited for large-scale SNN models in the presence of real-time constraints. Additionally, the present release adds new features, such as leaky-integrate-and-fire (LIF), 9-parameter Izhikevich, multi-compartment neuron models, and fourth order Runge Kutta integration.

A predictive mechanism in the brain enables primates to visually track a target with almost zero lag smooth pursuit eye movements, overcoming the delays in processing retinal inputs. Interestingly, it also allows pursuit of occluded targets with nonlinear motion patterns. We propose a recurrent neural network (RNN) model that rapidly learns the target velocity sequence and generates eye velocity signals to eliminate the initial lag between target and eye velocities, and to track occluded targets with nonlinear velocity. Moreover, the model is able to adapt to unpredictable perturbation and phase shift of target velocity and qualitatively reproduce the initial pursuit acceleration in experimentally observed timescales. We propose that the frontal eye field (FEF) region of the primate brain is homologous to the proposed RNN based on its persistent predictive activities during pursuit and location on the pursuit pathway.

Neuromorphic computing is a promising solution for reducing the size, weight and power of mobile embedded systems. In this paper, we introduce a realization of such a system by creating the first closed-loop battery-powered communication system between an IBM Neurosynaptic System (IBM TrueNorth chip) and an autonomous Android-Based Robotics platform. Using this system, we constructed a dataset of path following behavior by manually driving the Android-Based robot along steep mountain trails and recording video frames from the camera mounted on the robot along with the corresponding motor commands. We used this dataset to train a deep convolutional neural network implemented on the IBM NS1e board containing a TrueNorth chip of 4096 cores. The NS1e, which was mounted on the robot and powered by the robot's battery, resulted in a self-driving robot that could successfully traverse a steep mountain path in real time. To our knowledge, this represents the first time the IBM TrueNorth has been embedded on a mobile platform under closed-loop control.

Recent developments in neuromorphic engineering have enabled low-powered processing and sensing in robotics, leading to more efficient brain-like computation for many robotic tasks such as motion planning and navigation. However, present experiments in neuromorphic robotic systems have mostly been performed under controlled indoor settings, often with unlimited power supply. While this may be suitable for many applications, these algorithms often fail in outdoor dynamic environments that could benefit the most from the low size, weight, and power of neuromorphic devices. We present the current challenges of outdoor robotics, how current neuromorphic solutions can address these problems, our current approaches to the task, and what further needs to be achieved to create a complete neuromorphic solution to outdoor navigation and path planning.

A path planning algorithm is introduced that uses the timing of spiking neurons to create efficient routes. The algorithm is inspired by recent evidence showing activity-dependent plasticity of axon myelination after learning. Using this finding as inspiration, the algorithm's learning rule varies the simulated axon conductance velocity between neurons based on the relative cost of traversing the environment. In terms of path length and path cost, the spiking algorithm is as good or better than other path planners. However, the present spiking algorithm has the added advantage of adapting to change and context by altering axon delays in response to environmental experience. Because the spiking algorithm is suitable for implementation on neuromorphic hardware, it has the potential of realizing orders of magnitude gains in power efficiency and computational gains through parallelization, and thus should offer advantages for small, embedded systems.

Spiking neural network (SNN) models describe key aspects of neural function in a computationally efficient manner and have been used to construct large-scale brain models. Large-scale SNNs are challenging to implement, as they demand high-bandwidth communication, a large amount of memory, and are computationally intensive. Additionally, tuning parameters of these models becomes more difficult and time-consuming with the addition of biologically accurate descriptions. To meet these challenges, we have developed CARLsim 3, a user-friendly, GPU-accelerated SNN library written in C/C++ that is capable of simulating biologically detailed neural models. The present release of CARLsim provides a number of improvements over our prior SNN library to allow the user to easily analyze simulation data, explore synaptic plasticity rules, and automate parameter tuning. In the present paper, we provide examples and performance benchmarks highlighting the library's features.

Neuromorphic engineering is a fast growing field with great potential in both understanding the function of the brain, and constructing practical artifacts that build upon this understanding. For these novel chips and hardware to be useful, hardware compatible applications and simulation tools are needed. We argue that the neural circuit approach, in which networks of neuronal elements model brain circuitry are constructed, allows the development of practical applications and the exploration of brain function. At this level of abstraction, networks of 10 5 neurons or larger can be efficiently simulated, but still preserve the neuronal and synaptic dynamics that appear to be important for brain function. Because the neural circuit level supports spiking neural networks and the prevalent Addressable Event Representation (AER) communication scheme, it fits well with many existing neuromorphic hardware and simulation tools. To show how this approach can be applied, we present case studies of spiking neural networks in vision and recognition tasks based on one instantiation of a simulation environment. However, there are now many hardware options, simulation environments, and applications in this emerging field. These approaches and other considerations are discussed.

Neuromorphic engineering takes inspiration from biology to design brain-like systems that are extremely low-power, fault-tolerant, and capable of adaptation to complex environments. The design of these artificial nervous systems involves both the development of neuromorphic hardware devices and the development neuromorphic simulation tools. In this paper, we describe a simulation environment that can be used to design, construct, and run spiking neural networks (SNNs) quickly and efficiently using graphics processing units (GPUs). We then explain how the design of the simulation environment utilizes the parallel processing power of GPUs to simulate large-scale SNNs and describe recent modeling experiments performed using the simulator. Finally, we present an automated parameter tuning framework that utilizes the simulation environment and evolutionary algorithms to tune SNNs. We believe the simulation environment and associated parameter tuning framework presented here can accelerate the development of neuromorphic software and hardware applications by making the design, construction, and tuning of SNNs an easier task.

An evolutionary strategy (ES) algorithm was utilized to evolve a simulated neural network based on the known anatomy of the posterior parietal cortex (PPC), to perform a visually guided reaching task. In this task, a target remained visible for the duration of a trial, and an agent's goal was to move its hand to the target as rapidly as possible and remain for the duration of that trial. The ES was used to tune the strength of 15609 connections between neural areas and 4 parameters governing the neural dynamics. The model had sensory latencies replicating those found in recording studies with monkeys. The ES ran 100 times and generated very diverse networks that could all perform the task well. The evolved networks 1) showed velocity profiles consistent with biological movements, and 2) found solutions that reflect short-range excitation and long-range, contralateral inhibition similar to neurobiological networks. These results provide theoretical evidence for the important parameters and projections governing sensorimotor transformations in neural systems.

We present a novel neuromorphic robot that interacts through touch sensing and visual signaling on its surface. The robot's form factor is a convex, hemispheric shell containing trackballs for sensing touch, and LEDs for communication with users. In this paper, we explore tactile sensory decoding by constructing a spiking neural network (SNN) of somatosensory cortex. The SNN uses a biologically inspired, unsupervised learning rule, known as spike timing dependent plasticity, to classify a user's hand movements. In an evaluation of the network's ability to categorize hand movements, both rate and temporal neural coding performed well. Because of its unique form factor and means of interaction, this robot, which is called CARL-SJR, may be useful for exploring the neural coding of touch, and also for Human-Robot Interaction studies.

Spiking neural network (SNN) simulations with spike-timing dependent plasticity (STDP) often experience runaway synaptic dynamics and require some sort of regulatory mechanism to stay within a stable operating regime. Previous homeostatic models have used L1or L2normalization to scale the synaptic weights but the biophysical mechanisms underlying these processes remain undiscovered. We propose a model for homeostatic synaptic scaling that modifies synaptic weights in a multiplicative manner based on the average postsynaptic firing rate as observed in experiments. The homeostatic mechanism was implemented with STDP in conductance-based SNNs with Izhikevich-type neurons. In the first set of simulations, homeostatic synaptic scaling stabilized weight changes in STDP and prevented runaway dynamics in simple SNNs. During the second set of simulations, homeostatic synaptic scaling was found to be necessary for the unsupervised learning of V1 simple cell receptive fields in response to patterned inputs. STDP, in combination with homeostatic synaptic scaling, was shown to be mathematically equivalent to non-negative matrix factorization (NNMF) and the stability of the homeostatic update rule was proven. The homeostatic model presented here is novel, biologically plausible, and capable of unsupervised learning of patterned inputs, which has been a significant challenge for SNNs with STDP.

Attentional mechanisms allow the brain to enhance the representation and transmission of certain signals at the expense of others. The basal forebrain has been shown to play an important role in attention through its diverse set of interactions with sensory and associational areas. A recent empirical study indicates that the nucleus basalis, a subset of neurons located in the basal forebrain, is important for improving sensory processing by increasing reliability and decreasing redundancy in the cortex and thalamus [1, 2]. We developed a spiking neural network model that simulates the nucleus basalis' interaction with the thalamus and visual cortex. In this model, we simulated two modes of action by which it is thought that the nucleus basalis may be influencing sensory processing: (1) inhibitory projections from the nucleus basalis to the thalamic reticular nucleus, which disinhibit the lateral geniculate nucleus (LGN) and gate information into the cortex, and (2) cholinergic excitation of inhibitory neurons in the visual cortex. We showed that the inhibition of the thalamic reticular nucleus GABAergic neurons leads to an increase in the reliability of spikes in the LGN and cortex. We observed that a decrease in the burst to tonic firing ratio in the LGN, coupled with the cholinergic system increasing inhibition in the visual cortex caused decorrelation in the cortex. These findings will help us better understand the mechanisms behind the control of attention by the basal forebrain and shed light on how the orchestrated action of the basal forebrain on multiple target areas can improve information processing in the brain.

The brain's neuromodulatory systems play a key role in regulating decision-making and responding to environmental challenges. Attending to the appropriate sensory signal, filtering out noise, changing moods, and selecting behavior are all influenced by these systems. We introduce a neural network for action selection that is based on principles of neuromodulatory systems. The algorithm, which was tested on an autonomous robot, demonstrates valuable features such as fluid switching of behavior, gating in important sensory events, and separating signal from noise.

Biological neural systems are well known for their robust and power-efficient operation in highly noisy environments. We outline key modeling abstractions for the brain and focus on spiking neural network models. We discuss aspects of neuronal processing and computational issues related to modeling these processes. Although many of these algorithms can be efficiently realized in specialized hardware, we present a case study of simulation of the visual cortex using a GPU based simulation environment that is readily usable by neuroscientists and computer scientists and efficient enough to construct very large networks comparable to brain networks.

Game theory has been useful for understanding risk-taking, cooperation, and social behavior. However, in studies of the neural basis of decision-making during games of conflict, subjects typically play against an opponent with a predetermined strategy [1-3]. In the present study, human subjects played Hawk-Dove games against a neural agent, both simulated and robotic, with the ability to assess the potential costs and rewards of its actions and adapt its behavior accordingly. The neural agent's model was based on the assumption that the dopaminergic and serotonergic systems track expected rewards and costs, respectively [4]. The study consisted of two experimental days, one in which subjects' serotonin levels were lowered through acute tryptophan depletion (ATD), where human subjects played against neural agents whose simulated serotonin systems were altered as well. When the neural agent's serotonergic system was compromised, by turning off neural activity in its raphe nucleus, the neural agent tended towards aggressive behavior, due to its inability to assess the cost of its actions [4]. When subjects played against an aggressive neural agent, there was a significant shift in their strategy from Win-Stay-Lose-Shift (WSLS) to Tit-For-Tat (T4T). This shift to a T4T strategy may be similar to the rejection of unfair offers in the Ultimatum Game [2]. A T4T strategy, which is strategically less advantageous than WSLS, could send a message to another player that the subject believes he is being treated unfairly. In other studies, ATD led to increased defections in the Prisoner's Dilemma [3] and more rejections of offers in the Ultimatum Game [1]. In contrast, we did not observe a decrease of cooperativeness in our subjects due to ATD, but rather the emergence of a strongly significant shift in strategies based on opponent type. It may be that iterative interactions with a responsive, adaptive agent outweighed the effects of ATD in our human subjects. Additionally, the physical instantiation of the neural agent did not evoke stronger responses from subjects than did the simulated neural agent. We suggest that both the simulated and embodied versions of the neural agent evoked strong responses in subjects because of the neural agent's adaptive behavior. These results highlight the important interactions between human subjects and an agent that can adapt its behavior. Moreover, they reveal neuromodulatory mechanisms that give rise to cooperative and competitive behaviors.

Biological neural systems are well known for their robust and power-efficient operation in highly noisy environments. Biological circuits are made up of low-precision, unreliable and massively parallel neural elements with highly reconfigurable and plastic connections. Two of the most interesting properties of the neural systems are its self-organizing capabilities and its template architecture. Recent research in spiking neural networks has demonstrated interesting principles about learning and neural computation. Understanding and applying these principles to practical problems is only possible if large-scale spiking neural simulators can be constructed. Recent advances in low-cost multiprocessor architectures make it possible to build large-scale spiking network simulators. In this paper we review modeling abstractions for neural circuits and frameworks for modeling, simulating and analyzing spiking neural networks.

Neural network simulators that take into account the spiking behavior of neurons are useful for studying brain mechanisms and for engineering applications. Spiking neural network (SNN) simulators have been traditionally simulated on large-scale clusters, super-computers, or on dedicated hardware architectures. Alternatively, graphics processing units (GPUs) can provide a low-cost, programmable, and high-performance computing platform for simulation of SNNs. In this paper we demonstrate an efficient, Izhikevich neuron based large-scale SNN simulator that runs on a single GPU. The GPU-SNN model (running on an NVIDIA GTX-280 with 1 GB of memory), is up to 26 times faster than a CPU version for the simulation of 100 K neurons with 50 million synaptic connections, firing at an average rate of 7 Hz. For simulation of 100 K neurons with 10 million synaptic connections, the GPU-SNN model is only 1.5 times slower than real-time. Further, we present a collection of new techniques related to parallelism extraction, mapping of irregular communication, and compact network representation for effective simulation of SNNs on GPUs. The fidelity of the simulation results were validated against CPU simulations using firing rate, synaptic weight distribution, and inter-spike interval analysis. We intend to make our simulator available to the modeling community so that researchers will have easy access to large-scale SNN simulations.

A new RoboCup soccer league is being developed, focusing on human-robot interaction. In this league each team consists of both a human player, mounted on a Segway HT scooter, and a robotic version of the Segway; both human and robot players must cooperate to score goals. This paper details the design of our robotic Segway soccer brain-based device (SS-BBD). The SS-BBD control system is based on a large scale neural simulation, whose design is dictated by details from the published literature on vertebrate neuroanatomy, neurophysiology, and psychophysics. The physical device is completely autonomous, and possesses special manipulators for kicking and capturing a full-sized soccer ball. The SS-BBD uses visual and laser rangefinder information to recognize a variety of game related objects, which enables it to perform actions such as capturing the ball, kicking the ball to another player, shooting a goal, and maneuvering safely across the field. The SS-BBD can act autonomously or obey voice commands from the human player. This is an unprecedented level of human-robot teamwork on a soccer field, in that our players are not merely acting autonomously, but also communicate with each other and support each other on the field

The simultaneous study of brain function at all levels of organization is difficult to undertake with current experimental tools. Present day electrophysiology only allows the recording of at most hundreds of neurons while an animal is performing a behavioral task. Because of this limitation and the sheer complexity of the nervous system, computational modeling has become essential in developing theories of brain function. Accordingly, our group has constructed a series of brain-based devices (BBDs), that is, physical devices with simulated nervous systems that guide behavior, to serve as a heuristic for testing theories of brain function. Unlike animal models, BBDs permit analysis of activity at all levels of the nervous system as the device behaves in its environment. Although the principal focus of developing BBDs has been to test theories of brain function, this type of modeling may also provide a basis for robotic design and practical applications.

Whiskers are widely used by many animal species for navigation and texture discrimination. This paper describes Darwin IX, a mobile physical device equipped with artificial whiskers, the behavior of which is controlled by a neural simulation based on the rat somatosensory system. During its autonomous behavior, Darwin IX is able to discriminate among textures in its environment and learns to avoid textures that are paired with aversive events.

Our group has constructed a series of brain-based devices (BBDs); i.e. physical devices with simulated nervous systems that guide behavior, to serve as a heuristic for understanding brain function. Unlike conventional robots designed by engineering principles, BBDs are based on biological principles and alter their behavior to the environment through self-learning. The resulting systems autonomously generalize signals from the environment into perceptual categories and through adaptive behavior become increasingly successful in coping with the environment. Although the principal focus of developing BBDs has been to test theories of the nervous system, this approach may also provide a basis for robotic design and practical applications.

Sampling multisensory information and taking the appropriate motor action is critical for a biological organism's survival, but a difficult task for robots. We present a Neurally Organized Mobile Adaptive Device (NOMAD), whose behavior is controlled by a simulated nervous system based on the anatomy and physiology of the vertebrate brain, that is capable of action selection in a real world environment. NOMAD's nervous system consists of an auditory system, a visual system, a taste system, sets of motor neurons capable of triggering behavior, a tracking system driven by visual stimuli, and a value system. The device itself, which moves autonomously, has a CCD camera for vision, microphones for hearing, and a gripper manipulator to pick up and taste objects by measuring the object's conductivity. Similar to a biological organism, NOMAD learns to categorize sensory information from its environment with no prior instruction, associate positive and negative value with this sensory information, and then learn to select the appropriate motor actions. We suggest that this neurobiological approach to action selection may be generalized to other robot systems.