As robots become smarter and more ubiquitous, optimizing the power consumption of intelligent compute becomes imperative towards ensuring the sustainability of technological advancements. Neuromorphic computing hardware makes use of biologically inspired neural architectures to achieve energy and latency improvements compared to conventional von Neumann computing architecture. Applying these benefits to robots has been demonstrated in several works in the field of neurorobotics, typically on relatively simple control tasks. Here, we introduce an example of neuromorphic computing applied to the real-world industrial task of object insertion. We trained a spiking neural network (SNN) to perform force-torque feedback control using a reinforcement learning approach in simulation. We then ported the SNN to the Intel neuromorphic research chip Loihi interfaced with a KUKA robotic arm. At inference time we show latency competitive with current CPU/GPU architectures, and one order of magnitude less energy usage in comparison to traditional low-energy edge-hardware. We offer this example as a proof of concept implementation of a neuromoprhic controller in real-world robotic setting, highlighting the benefits of neuromorphic hardware for the development of intelligent controllers for robots.

Humans and animals learn throughout their lives from limited amounts of sensed data, both with and without supervision. Autonomous, intelligent robots of the future are often expected to do the same. The existing continual learning (CL) methods are usually not directly applicable to robotic settings: they typically require buffering and a balanced replay of training data. A few-shot online continual learning (FS-OCL) setting has been proposed to address more realistic scenarios where robots must learn from a non-repeated sparse data stream. To enable truly autonomous life-long learning, an additional challenge of detecting novelties and learning new items without supervision needs to be addressed. We address this challenge with our new prototype-based approach called Continually Learning Prototypes (CLP). In addition to being capable of FS-OCL learning, CLP also detects novel objects and learns from them without supervision. To mitigate forgetting, CLP utilizes a novel metaplasticity mechanism that adapts the learning rate individually per prototype. CLP is rehearsal-free, hence does not require a memory buffer, and is compatible with neuromorphic hardware, characterized by ultra-low power consumption, real-time processing abilities, and on-chip learning. Indeed, we have open-sourced both the PyTorch implementation of CLP and a simpler version in the neuromorphic software framework Lava, targetting Intel’s neuromorphic chip Loihi 2. We evaluate CLP on a robotic vision dataset, OpenLORIS. In a low-instance FS-OCL scenario, CLP shows state-of-the-art results. In the open world, CLP detects novelties with superior precision and recall and learns features of the detected novel classes without supervision, achieving a strong baseline of 99% base class and 65%/76% (5-shot/10-shot) novel class accuracy.

Event-based vision sensors achieve up to three orders of magnitude better speed vs. power consumption trade off in high-speed control of UAVs compared to conventional image sensors. Event-based cameras produce a sparse stream of events that can be processed more efficiently and with a lower latency than images, enabling ultra-fast vision-driven control. Here, we explore how an event-based vision algorithm can be implemented as a spiking neuronal network on a neuromorphic chip and used in a drone controller. We show how seamless integration of event-based perception on chip leads to even faster control rates and lower latency. In addition, we demonstrate how online adaptation of the SNN controller can be realised using on-chip learning. Our spiking neuronal network on chip is the first example of a neuromorphic vision-based controller on chip solving a high-speed UAV control task. The excellent scalability of processing in neuromorphic hardware opens the possibility to solve more challenging visual tasks in the future and integrate visual perception in fast control loops.

Deep artificial neural networks apply principles of the brain's information processing that led to breakthroughs in machine learning spanning many problem domains. Neuromorphic computing aims to take this a step further to chips more directly inspired by the form and function of biological neural circuits, so they can process new knowledge, adapt, behave, and learn in real time at low power levels. Despite several decades of research, until recently, very few published results have shown that today's neuromorphic chips can demonstrate quantitative computational value. This is now changing with the advent of Intel's Loihi, a neuromorphic research processor designed to support a broad range of spiking neural networks with sufficient scale, performance, and features to deliver competitive results compared to state-of-the-art contemporary computing architectures. This survey reviews results that are obtained to date with Loihi across the major algorithmic domains under study, including deep learning approaches and novel approaches that aim to more directly harness the key features of spike-based neuromorphic hardware. While conventional feedforward deep neural networks show modest if any benefit on Loihi, more brain-inspired networks using recurrence, precise spike-timing relationships, synaptic plasticity, stochasticity, and sparsity perform certain computation with orders of magnitude lower latency and energy compared to state-of-the-art conventional approaches. These compelling neuromorphic networks solve a diverse range of problems representative of brain-like computation, such as event-based data processing, adaptive control, constrained optimization, sparse feature regression, and graph search.

Neuromorphic hardware offers computing platforms for the efficient implementation of spiking neural networks (SNNs) that can be used for robot control. Here, we present such an SNN on a neuromorphic chip that solves a number of tasks related to simultaneous localization and mapping (SLAM): forming a map of an unknown environment and, at the same time, estimating the robot's pose. In particular, we present an SNN mechanism to detect and estimate errors when the robot revisits a known landmark and updates both the map and the path integration speed to reduce the error. The whole system is fully realized in a neuromorphic device, showing the feasibility of a purely SNN-based SLAM, which could be efficiently implemented in a small form-factor neuromorphic chip.

Spiking Neuronal Networks (SNNs) realized in neuromorphic hardware lead to low-power and low-latency neuronal computing architectures. Neuromorphic computing systems are most efficient when all of perception, decision making, and motor control are seamlessly integrated into a single neuronal architecture that can be realized on the neuromorphic hardware. Many neuronal network architectures address the perception tasks, while work on neuronal motor controllers is scarce. Here, we present an improved implementation of a neuromorphic PID controller. The controller was realized on Intel's neuromorphic research chip Loihi and its performance tested on a drone, constrained to rotate on a single axis. The SNN controller is built using neuronal populations, in which a single spike carries information about sensed and control signals. Neuronal arrays perform computation on such sparse representations to calculate the proportional, derivative, and integral terms. The SNN PID controller is compared to a PID controller, implemented in software, and achieves a comparable performance, paving the way to a fully neuromorphic system in which perception, planning, and control are realized in an on-chip SNN.

We present a spiking neural network (SNN) for visual pattern recognition with on-chip learning on neuromorphic hardware. We show how this network can learn simple visual patterns composed of horizontal and vertical bars sensed by a Dynamic Vision Sensor, using a local spike-based plasticity rule. During recognition, the network classifies the pattern's identity while at the same time estimating its location and scale. We build on previous work that used learning with neuromorphic hardware in the loop and demonstrate that the proposed network can properly operate with on-chip learning, demonstrating a complete neuromorphic pattern learning and recognition setup. Our results show that the network is robust against noise on the input (no accuracy drop when adding 130% noise) and against up to 20% noise in the neuron parameters.

The Lobula giant movement detector (LGMD) is an identified neuron of the locust that detects looming objects and triggers the insect's escape responses. Understanding the neural principles and network structure that leads to these fast and robust responses can facilitate the design of efficient obstacle avoidance strategies for robotic applications. Here, we present a neuromorphic spiking neural network model of the LGMD driven by the output of a neuromorphic dynamic vision sensor (DVS), which incorporates spiking frequency adaptation and synaptic plasticity mechanisms, and which can be mapped onto existing neuromorphic processor chips. However, as the model has a wide range of parameters and the mixed-signal analog-digital circuits used to implement the model are affected by variability and noise, it is necessary to optimize the parameters to produce robust and reliable responses. Here, we propose to use differential evolution (DE) and Bayesian optimization (BO) techniques to optimize the parameter space and investigate the use of self-adaptive DE (SADE) to ameliorate the difficulties of finding appropriate input parameters for the DE technique. We quantify the performance of the methods proposed with a comprehensive comparison of different optimizers applied to the model and demonstrate the validity of the approach proposed using recordings made from a DVS sensor mounted on an unmanned aerial vehicle (UAV).

Neuromorphic computing is a new paradigm for design of both the computing hardware and algorithms inspired by biological neural networks. The event-based nature and the inherent parallelism make neuromorphic computing a promising paradigm for building efficient neural network based architectures for control of fast and agile robots. In this paper, we present a spiking neural network architecture that uses sensory feedback to control rotational velocity of a robotic vehicle. When the velocity reaches the target value, the mapping from the target velocity of the vehicle to the correct motor command, both represented in the spiking neural network on the neuromorphic device, is autonomously stored on the device using on-chip plastic synaptic weights. We validate the controller using a wheel motor of a miniature mobile vehicle and inertia measurement unit as the sensory feedback and demonstrate online learning of a simple “inverse model” in a two-layer spiking neural network on the neuromorphic chip. The prototype neuromorphic device that features 256 spiking neurons allows us to realise a simple proof of concept architecture for the purely neuromorphic motor control and learning. The architecture can be easily scaled-up if a larger neuromorphic device is available.

In this paper, we investigate the use of ultra low-power, mixed signal analog/digital neuromorphic hardware for implementation of biologically inspired neuronal path integration and map formation for a mobile robot. We perform spiking network simulations of the developed architecture, interfaced to a simulated robotic vehicle. We then port the neuronal map formation architecture on two connected neuromorphic devices, one of which features on-board plasticity, and demonstrate the feasibility of a neuromorphic realization of simultaneous localization and mapping (SLAM).

Simultaneous localization and mapping (SLAM) is one of the core tasks of mobile autonomous robots. Looking for power efficient and embedded solutions for SLAM is an important challenge when building controllers for small and agile robots. Biological neural systems of even simple animals are until now unprecedented in their ability to localize themselves in an unknown environment. Neuromorphic engineering offers ultra low-power and compact computing hardware, in which biologically inspired neuronal architectures for SLAM can be realised. In this paper, we propose an on chip approach for one of the components of SLAM: path integration. Our solution takes inspiration from biology and uses motor command information to estimate the orientation of an agent solely in a spiking neural network. We realise this network on a neuromorphic device that implements artificial neurons and synapses with analog electronics. The neural network receives visual input from an event-based camera and uses this information to correct the on-chip spiking neurons estimate of the robot's orientation. This system can be easily integrated with other localization and mapping components on chip and is a step towards a fully neuromorphic SLAM.

The problem of inferring distances from a visual sensor to objects in a scene - referred to as depth estimation - can be solved in various ways. Among those, stereo vision is a method in which two sensors observe the same scene from different viewpoints. To recover the three-dimensional coordinates of a point, its two projections - one in each view - can be used for triangulation. However, the pair of points in the two views that correspond to each other has to be found first. This is known as stereo-matching and is usually a computationally expensive operation. Traditionally, this is performed by describing a point in the first view with some information from its surrounding, e.g. in a feature vector, and then searching for a match with a point described in a similar way in the other view. In this work, we propose a simple idea that alleviates this stereo-matching problem using an active component: a mirror-galvanometer driven laser. The laser beam is deflected by actuating two mirrors, thus creating a sequence of "light spots" in the scene. At these spots, contrast changes quickly. We capture those contrast changes by two Dynamic Vision Sensors (DVS). The high time-resolution of these sensors enables the detection of the laser-induced events in time and their matching using lightweight computation. This method enables event-based depth estimation at a high speed, low computational cost, and without exact sensor synchronization.

We demonstrate a system that can reconstruct the three-dimensional structure of a scene using two Dynamic Vision Sensors (DVS) and an active component: a Mirror Galvanometer-driven Laser (MGDL). Our system uses two concurrent methods to estimate depth: 1) triangulation from the two DVS cameras calibrated as stereo rig, where stereo matching is greatly simplified by actively generating events in the scene with the laser beam, and 2) a technique derived from structured light, where we send the laser beam in a known direction and detect its projection in each of the two cameras. For the demonstration, we add two interactive components: the user can trigger (re)scanning of a specific location in the scene by pointing a hand-held laser pointer to it, and virtual objects can be rendered with correct scaling and distance on a screen using the reconstructed depth.

This special issue presents perspectives on learning and adaptation as they apply to a number of cognitive phenomena including pupil dilation in humans and attention in robots, natural language acquisition and production in embodied agents (robots), human-robot game play and social interaction, neural-dynamic modelling of active perception and neural-dynamic modelling of infant development in the Piagetian A-not-B task. The aim of the special issue, through its contributions, is to highlight some of the critical neural-dynamic and behavioural aspects of learning as it grounds adaptive responses in robotic- and neural-dynamic systems.

We demonstrate a 3D imaging system that produces sparse depth maps. It consists in a liquid focus-tunable lens whose focal power can be changed at high speed, placed in front of a SCAMP5 vision-chip embedding processing capabilities in each pixel. The focus-tunable lens performs focal sweeps with shallow depth of fields. These are sampled by the vision chip taking multiple images at different focus and analyzed on-chip to produce a single depth frame. The combination of the focus tunable-lens with the vision-chip, enabling near-focal plane processing, allows us to present a compact passive system that is static, monocular, real-time (> 25FPS) and low-power (<; 1.6W).

Event based sensors and neural processing architectures represent a promising technology for implementing low power and low latency robotic control systems. However, the implementation of robust and reliable control architectures using neuromorphic devices is challenging, due to their limited precision and variable nature of their underlying computing elements. In this paper we demonstrate robust obstacle avoidance and target acquisition behaviors in a compact mobile platform controlled by a neuromorphic sensory-processing system and validate its performance in a number of robotic experiments.

We present a neuromorphic adaptation of a spiking neural network model of the locust Lobula Giant Movement Detector (LGMD), which detects objects increasing in size in the field of vision (looming) and can be used to facilitate obstacle avoidance in robotic applications. Our model is constrained by the parameters of a mixed signal analog-digital neuromorphic device, developed by our group, and is driven by the output of a neuromorphic vision sensor. We demonstrate the performance of the model and how it may be used for obstacle avoidance on an unmanned areal vehicle (UAV).

A computing architecture based on neuronal principles is presented, which implements learning to reach towards visually-perceived targets for an embodied agent. The whole behavioural loop from object perception to motor control is realised in the architecture using attractor dynamics and Dynamic Neural Fields. The sensory-motor mappings, involved in generation of saccadic gaze shifts and goal-directed arm movements, adapt in the system autonomously during the behaviour. A network of neural-dynamic nodes organises activation and deactivation of the behavioural modules of the architecture, leading to an autonomous process model of learning to look and to reach. The architecture was implemented and validated on a simulated robot.

Increasingly widespread available haptic sensors mounted on articulated hands offer new sensory channels that can complement shape extraction from vision to enable a more robust handling of objects in cases when vision is restricted or even unavailable. However, to estimate object shape from haptic interaction data is a difficult challenge due to the complexity of the contact interaction between the movable object and sensor surfaces, leading to a coupled estimation problem of shape and object pose. While for vision efficient solutions to the underlying SLAM problem are known, the available information is much sparser in the tactile case, posing great difficulties for a straightforward adoption of standard SLAM algorithms. In the present paper, we thus explore whether a biologically inspired model based on dynamic neural fields can offer a route towards a practical algorithm for tactile SLAM. Our study is focused on a restricted scenario where a two-fingered robot hand manipulates an n-gon with a fixed rotational axis. We demonstrate that our model can accumulate shape information from reasonably short interaction sequences and autonomously build a representation despite significant ambiguity of the tactile data due to the rotational periodicity of the object. We conclude that the presented framework may be a suitable basis to solve the tactile SLAM problem also in more general settings which will be the focus of subsequent work.

We introduce a dynamic neural algorithm called Dynamic Neural (DN) SARSA(λ) for learning a behavioral sequence from delayed reward. DN-SARSA(λ) combines Dynamic Field Theory models of behavioral sequence representation, classical reinforcement learning, and a computational neuroscience model of working memory, called Item and Order working memory, which serves as an eligibility trace. DN-SARSA(λ) is implemented on both a simulated and real robot that must learn a specific rewarding sequence of elementary behaviors from exploration. Results show DN-SARSA(λ) performs on the level of the discrete SARSA(λ), validating the feasibility of general reinforcement learning without compromising neural dynamics.

Robotic agents that interact with humans and perform complex, everyday tasks in natural environments will require a system to autonomously organize their behavior. Current systems for robotic behavioral organization typically abstract from the low-level sensory-motor embodiment of the robot, leading to a gap between the level at which a sequence of actions is planned and the levels of perception and motor control. This gap is a major bottleneck for the autonomy of systems in complex, dynamic environments. To address this issue, we present a neural-dynamic framework for behavioral organization, in which the action selection mechanism is tightly coupled to the agent's sensory-motor systems. The elementary behaviors (EBs) of the robot are dynamically organized into sequences based on task-specific behavioral constraints and online perceptual information. We demonstrate the viability of our approach by implementing a neural-dynamic architecture on the humanoid robot NAO. The system is capable of producing sequences of EBs that are directed at objects (e.g., grasping and pointing). The sequences are flexible in that the robot autonomously adapts the individual EBs and their sequential order in response to changes in the sensed environment. The architecture can accommodate different tasks and can be articulated for different robotic platforms. Its neural-dynamic substrate is particularly well-suited for learning and adaptation.

Spatial language is a privileged channel of human-robot interaction. Here, we extend a neural-dynamic architecture for grounded spatial language in three ways. First, we introduce autonomous selection between viewer-centered and intrinsic reference frames, using an estimation of the reference object orientation to determine its intrinsic axes. Second, we employ an orientation estimation dynamics to represent the configurations of reference objects for spatial terms such as “between”. Third, we enhance the autonomy of the system so that the required sequence of attentional shifts, coordinate transforms, and selection decisions emerges from the time-continuous neural dynamics. In a robotic implementation we demonstrate how spatial language may be grounded in simple feature information obtained from video cameras and applied flexibly to dynamical scenes.

For an autonomous robotic system, the ability to share the same workspace and interact with humans is the basis for cooperative behavior. In this work, we investigate human spatial language as the communicative channel between the robot and the human, facilitating their joint work on a tabletop. We specifically combine the theory of Dynamic Neural Fields that represent perceptual and cognitive states with motor control and linguistic input in a robotic demonstration. We show that such a neural dynamic framework can integrate across symbolic, perceptual, and motor processes to generate task-specific spatial communication in real time.