The number of deployed artificial intelligence (AI) models is growing rapidly. Large language models (LLMs) are both compute and memory intensive in training and inference, regularly having billions of parameters and requiring petabytes of training data. Neuromorphic computing seeks to improve scaling in machine learning by bypassing the memory bandwidth limitations inherent in hardware that LLMs are currently run on, by designing biologically-inspired systems. Neuromorphic algorithm-hardware co-design requires reproducible low-level, hardware-optimised training and testing inputs. In this work, we propose a framework to generate the first high-resolution, multimodal, phonetically-rich neuromorphic eventset, including a novel synchronisation signal. This overcomes the limitations of datasets by encoding information in the form of events, a low-level representation that capture the underlying physical phenomenon with high fidelity while remaining sparse, compressed and digital.We show that events are highly compressible and that datasets recorded with traditional methods would require more than 100× more memory to capture the same level of temporal granularity.

This paper provides an overview of the ongoing work to enable novel modalities of passive monocular neuromorphic vision in the NimbleAI sensing-processing architecture; namely, foveated and light-field event-driven vision with selective visual attention. The latter vision modality encodes 3D visual surroundings as sparse visual events in a 4D spatiotemporal domain, adding depth to current representation of visual information delivered by Dynamic Vision Sensors (DVS). The NimbleAI architecture implements hardware support for efficient execution of mainstream computer vision algorithms and AI models using these visual inputs. The architecture is designed to harness the latest advancements in 3D silicon integration, making it possible to squeeze sensing and spiking circuitry, memory, and processing engines into a miniature silicon volume.

The NimbleAI Horizon Europe project leverages key principles of energy-efficient visual sensing and processing in biological eyes and brains, and harnesses the latest advances in$\mathbf{33D}$stacked silicon integration, to create an integral sensing-processing neuromorphic architecture that efficiently and accurately runs computer vision algorithms in area-constrained endpoint chips. The rationale behind the NimbleAI architecture is: sense data only with high information value and discard data as soon as they are found not to be useful for the application (in a given context). The NimbleAI sensing-processing architecture is to be specialized after-deployment by tunning system-level trade-offs for each particular computer vision algorithm and deployment environment. The objectives of NimbleAI are: (1)$\mathbf{100x}$performance per mW gains compared to state-of-the-practice solutions (i.e., CPU/GPUs processing frame-based video); (2)$\mathbf{50x}$processing latency reduction compared to CPU/GPUs; (3) energy consumption in the order of tens of mWs; and (4) silicon area of approx. 50 mm2.