In the realms of professional and semi-professional sports across various major leagues, the role of a scouter, scout, or talent scout is pivotal. This individual attends games to analyze and collect comprehensive data about the events unfolding during the match. Traditionally, scouters rely on scouting software where they input match data, which then generates extensive statistics, conclusions, and recommendations. These insights are subsequently utilized by sports clubs for purposes such as player recruitment and devising strategies against specific rivals. This research introduces a novel line of inquiry, harnessing AI/ML (Artificial Intelligence/Machine Learning) techniques, predominantly convolutional neural networks, to automate the collection of data from physical sports events. This includes tracking the movement of the ball, the positioning of players, the arm used for hitting the ball, and more. The objective is to advance current sports scouting systems, which predominantly depend on manual data collection, towards a more automated, accurate, and efficient methodology. Utilizing nine tennis match videos from YouTube covering three different court surfaces (hard court, grass court, and clay court), this study has achieved high effectiveness in detecting players (99.3%), their body joints (100%), the tennis ball (94.13%), the segments of the tennis court ( ∼ 100%), and identifying the type of stroke (forehand or backhand) (100%). Moreover, it successfully extracts detailed statistics, such as the number of points played, stroke effectiveness, and efficiency, all categorized by specific court areas, culminating in the creation of a heat map. This advancement not only streamlines the scouting process but also offers richer, data-driven insights into player performance and game dynamics.

Latency and computational cost often limit the use of Nonlinear Model Predictive Control (NMPC) in real-time robotics. To address this limitation, our work investigates FPGA-implemented Neural Controllers (NC) trained through supervised learning, mimicking NMPC. We show that inexpensive embedded FPGA hardware is sufficient to implement these neural controllers for high-frequency control of robotic systems. We demonstrate kilohertz control rates for a cartpole and offload control to the FPGA hardware on the F1TENTH race car. The FPGA NC outperforms NMPC on the cartpole, due to the faster control rate afforded by faster NC inference. The code and hardware implementation for this paper are available at https://github.com/SensorsINI/Neural-Control-Tools.

Great interest is directed toward real-time applications to determine the exact location of sensor nodes deployed in an area of interest. In this paper, we present a novel approach using a combination of the Kalman filter and regularized bounding box method for localizing unknown nodes in an area using an FPGA-enhanced edge computing UAV whose trajectory is known and is represented as the position of many anchors. The UAV is equipped with a GPS system that allows it to gather location data of sensor nodes as it moves around its environment. We employ a regularized bounding box to predict the positions of the unknown nodes using regularization factors and we use the Kalman filter algorithm to smooth and improve the accuracy of the sensor nodes to be localized. In order to localize the unknown nodes, the UAV receives the number of hops from each node and uses this information as input to the localization algorithm. Furthermore, the use of an FPGA board allows for real-time processing of sensory data, enabling the UAV to make fast and accurate decisions in dynamic environments. The localization algorithm was implemented on the FPGA board “Zynq MiniZed 7007s evaluation board” using Xilinx blocks in Simulink, and the generated code was converted into VHDL using Xilinx System Generator. The algorithm was simulated and synthesized using “Vivado” software. In fact, the proposed system was evaluated by comparing the performances achieved through two different implementations: Hardware and Software implementation. In effect, the performance of FPGA hardware implementation presents a new achievement in localization due to its easy testing and fast implementation. Our results show that this approach can efficiently locate unknown nodes with good latency and high accuracy. In fact, the execution time of the FPGA-integrated algorithm is reduced by about 60 times compared to the software implementation and the power consumption is about 100 mW, which proves the suitability of FPGA for localization in WSNs, offering a promising solution for various mobile WSN applications.

Abstract As localization represents the main core of various wireless sensor network applications, several localization algorithms have been suggested in wireless sensor network research. In this article, we put forward an iterative bounding box algorithm enhanced by a Kalman filter to refine the unknown node’s estimated position. In fact, several research efforts are currently in progress to extend the 2D positioning algorithm in WSNs to 3D that reflects reality and the most practical applications. Subsequently, we replace a large number of GPS-equipped anchors with a single mobile anchor. In our studies, we consider the type of range-free sensor network exploiting the wireless sensors connectivity. We assess the performance of our algorithm using exhaustive experiments on several isotropic and anisotropic topologies. Our proposed algorithm can fulfill the joint goals of algorithm transparency and accuracy for various scenarios by evaluating parameters such as localization accuracy whilst changing other simulation parameters such as the effect of communication range, mobile anchor node position and sensor node deployment topology. It has been proven by the results of the experiments that the proposed algorithm effectively reduces the location error without requiring more equipment or increasing the communication cost.

This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

Abstract Neuromorphic computation processes sensors output in the spiking domain, which presents constraints in many cases when converting information to spikes, loosing, as example, temporal accuracy. This paper presents a spike-based system to adapt audio information from low-power pulse-density modulation (PDM) microelectromechanical systems microphones into rate coded spike frequencies. These spikes could be directly used by the neuromorphic auditory sensor (NAS) for frequency decomposition in different bands, avoiding the analog or digital conversion to spike streams. This improves the time response of the NAS, allowing its use in more time restrictive applications. This adaptation was conducted in VHDL as an interface for PDM microphones, converting their pulses into temporal distributed spikes following a pulse-frequency modulation scheme with an accurate inter-spike-interval, known as PDM to spikes interface (PSI). We introduce a new architecture of spike-based band-pass filter to reject DC components and distribute spikes in time. This was tested in two scenarios, first as a stand-alone circuit for its characterization, and then integrated with a NAS for verification. The PSI achieves a total harmonic distortion of $$-$$ - 46.18 dB and a signal-to-noise ratio of 63.47 dB, demands less than 1% of the resources of a Spartan-6 FPGA and its power consumption is around 7 mW.

Biological nervous systems typically perform the control of numerous degrees of freedom for example in animal limbs. Neuromorphic engineers study these systems by emulating them in hardware for a deeper understanding and its possible application to solve complex problems in engineering and robotics. Central-Pattern-Generators (CPGs) are part of neuro-controllers, typically used at their last steps to produce rhythmic patterns for limbs movement. Different patterns and gaits typically compete through winner-take-all (WTA) circuits to produce the right movements. In this work we present a WTA circuit implemented in a Spiking-Neural-Network (SNN) processor to produce such patterns for controlling a robotic arm in real-time. The robot uses spike-based proportional-integrative-derivative (SPID) controllers to keep a commanded joint position from the winner population of neurons of the WTA circuit. Experiments demonstrate the feasibility of robotic control with spiking circuits following brain-inspiration.

In this work, an all-in-one neuromorphic controller system with reduced latency and power consumption for a robotic arm is presented. Biological muscle movement consists of stretching and shrinking fibres via spike-commanded signals that come from motor neurons, which in turn are connected to a central pattern generator neural structure. In addition, biological systems are able to respond to diverse stimuli rather fast and efficiently, and this is based on the way information is coded within neural processes. As opposed to human-created encoding systems, neural ones use neurons and spikes to process the information and make weighted decisions based on a continuous learning process. The Event-Driven Scorbot platform (ED-Scorbot) consists of a 6 Degrees of Freedom (DoF) robotic arm whose controller implements a Spiking Proportional-Integrative-Derivative algorithm, mimicking in this way the previously commented biological systems. In this paper, we present an infrastructure upgrade to the ED-Scorbot platform, replacing the controller’s hardware, which was comprised of two Spartan Field Programmable Gate Arrays (FPGAs) and a barebone computer, with an edge device, the Xilinx Zynq-7000 SoC (System on Chip) which reduces the response time, power consumption and overall complexity.

Membrane computing is an unconventional computing paradigm that has gained much attention in recent decades because of its massively parallel character and its usefulness to build models of complex systems. However, until now, there was no generic hardware implementation of P systems. Computational frameworks to execute P systems up to this day rely on the simulation of the parallel working mechanisms of P systems by inherently sequential algorithms. Such algorithms can then be implemented as is or can be parallelized, up to a certain point, to run on parallel computers. However, this is not as efficient as a dedicated parallel hardware implementation. There have been ad hoc implementations of particular P systems for parallel hardware, but they lack to be problem-generic or they are not scalable enough to implement large P systems. In this paper, a first intrinsically parallel hardware architecture to implement generic P system models is introduced. It is designed to be straightforwardly implemented in programmable logic circuits like FPGAs. The feasibility and correct execution of our architecture has been verified by means of a simulator, and several simulation results for different P system examples have been analysed to foresee the pros and cons of this design.

The brain cortex processes visual information to classify it following a scheme that has been mimicked by Convolutional Neural Networks (CNN). Specialised hardware accelerators are currently used as CPU co-processors for mobile applications. These accelerators are getting closer to the sensors for an edge computation of its output towards a faster and lower power consumption improvements. In this demonstration we use a dynamic vision sensor (inspired in the retina neural cells) as a visual source of the NullHop CNN accelerator deployed on a MPSoC FPGA and placed into a mobile robot for edge-computing the visual information and classify it to properly command a Summit-XL mobile robot for a target destiny. The reduced latency of the used CNN accelerator allows to process several histograms before taking a movement decision. A distance sensor mounted on the robot ensures that the direction change is done at the right distance for a proper path following.

Spiking convolutional neural networks have become a novel approach for machine vision tasks, due to the latency to process an input stimulus from a scene, and the low power consumption of these kind of solutions. Event-based systems only perform sum operations instead of sum of products of frame-based systems. In this work an upgrade of a neuromorphic event-based convolution accelerator for SCNN, which is able to perform multiple layers with different kernel sizes, is presented. The system has a latency per layer from 1.44 us to 9.98us for kernel sizes from 1x1 to 7x7.

Along with the proliferation of mobile devices and wireless signal coverage, IoT devices, such as smart wristbands \nfor monitoring its owner’s activity or sleep patterns, get great popularity. Wearable technology in human \nlife has become quite useful due to the information given (sleep hours, heart rate, etc). However, wearables \nfor animals does not give information about behaviour directly: they collect raw data that is sent to a server \nwhere, after a post-processing step, the behaviour is known. In this work, we present a smart IoT device that \nclassifies different animal behaviours from the information obtained from on-board sensors using an embedded \nneural network running in the device. This information is uploaded to a server through a wireless sensor \nnetwork based on Zigbee communication. The architecture of the device allows an easy assembly in a reduced \ndimension wearable case. The firmware allows a modular functionality by activating or deactivating \nmodules independently, which improve the power efficiency of the device. The power consumption has been \nanalyzed, allowing the 1Ah battery to work the system during several days. A novel localization and distance \nestimation technique (for 802.15.4 networks) is presented to recover a lost device in Do˜nana National Park \nwith unidirectional antennas and log-normalization distance estimation over RSSI.

In this demonstration a spiking neural network architecture for vision recognition using an FPGA spiking convolution processor, based on leaky integrate and fire neurons (LIF) and a SpiNNaker board is presented. The network has been trained with Poker-DVS dataset in order to classify the four different card symbols. The spiking convolution processor extracts features from images in form of spikes, computes by one layer of 64 convolutions. These features are sent to an OKAERtool board that converts from AER to 2-7 protocol to be classified by a spiking neural network deployed on a SpiNNaker platform.

Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving many state-of-the-art (SOA) visual processing tasks. Even though graphical processing units are most often used in training and deploying CNNs, their power efficiency is less than 10 GOp/s/W for single-frame runtime inference. We propose a flexible and efficient CNN accelerator architecture called NullHop that implements SOA CNNs useful for low-power and low-latency application scenarios. NullHop exploits the sparsity of neuron activations in CNNs to accelerate the computation and reduce memory requirements. The flexible architecture allows high utilization of available computing resources across kernel sizes ranging from 1×1 to 7×7. NullHop can process up to 128 input and 128 output feature maps per layer in a single pass. We implemented the proposed architecture on a Xilinx Zynq field-programmable gate array (FPGA) platform and presented the results showing how our implementation reduces external memory transfers and compute time in five different CNNs ranging from small ones up to the widely known large VGG16 and VGG19 CNNs. Postsynthesis simulations using Mentor Modelsim in a 28-nm process with a clock frequency of 500 MHz show that the VGG19 network achieves over 450 GOp/s. By exploiting sparsity, NullHop achieves an efficiency of 368%, maintains over 98% utilization of the multiply-accumulate units, and achieves a power efficiency of over 3 TOp/s/W in a core area of 6.3 mm2. As further proof of NullHop's usability, we interfaced its FPGA implementation with a neuromorphic event camera for real-time interactive demonstrations.

Neural networks algorithms are commonly used to recognize patterns from different data sources such as audio or vision. In image recognition, Convolutional Neural Networks are one of the most effective techniques due to the high accuracy they achieve. This kind of algorithms require billions of addition and multiplication operations over all pixels of an image. However, it is possible to reduce the number of operations using other computer vision techniques rather than frame-based ones, e.g. neuromorphic frame-free techniques. There exists many neuromorphic vision sensors that detect pixels that have changed their luminosity. In this study, an event-based convolution engine for FPGA is presented. This engine models an array of leaky integrate and fire neurons. It is able to apply different kernel sizes, from 1x1 to 7x7, which are computed row by row, with a maximum number of 64 different convolution kernels. The design presented is able to process 64 feature maps of 7x7 with a latency of 8.98μs.

In this demonstration we present a spiking neural network architecture for audio samples classification using SpiNNaker. The network consists of different leaky integrate-and-fire neuron layers. The connections between them are trained using firing rate based algorithms. Tests use sets of pure tones with frequencies that range from 130.813 to 1396.91 Hz. Audio signals coming from the computer are converted to spikes using a Neuromorphic Auditory Sensor and, after that, this information is sent to the SpiNNaker board through a PCB that translates from AER to 2-of-7 protocol. The classification output obtained in the spiking neural network deployed on SpiNNaker is then shown in the computer screen. Different levels of random noise are added to the original audio signals in order to test the robustness of the classification system.

This paper describes the software and FPGA implementation of a Retinal Ganglion Cell model which detects moving objects. It is shown how this processing, in conjunction with a Dynamic Vision Sensor as its input, can be used to extrapolate information about object position. Software-wise, a system based on an array of these of RGCs has been developed in order to obtain up to two trackers. These can track objects in a scene, from a still observer, and get inhibited when saccadic camera motion happens. The entire processing takes on average 1000 ns/event. A simplified version of this mechanism, with a mean latency of 330 ns/event, at 50 MHz, has also been implemented in a Spartan6 FPGA.

This demonstration shows how object detection and tracking are possible thanks to a new implementation which takes inspiration from the visual processing of a particular type of ganglion cell in the retina.

Multisensory integration is commonly used in various robotic areas to collect much more information from an environment using different and complementary types of sensors. This demonstration presents a scenario where the motor rotation frequency is obtained using an AER DVS128 retina chip (Dynamic Vision Sensor) and a frequency decomposer auditory system on a FPGA that mimics a biological cochlea. Both of them are spike-based sensors with Address-Event-Representation (AER) outputs. A new AER monitor hardware interface, based on a Spartan-6 FPGA, allows two operational modes: real-time (up to 5 Mevps through USB2.0) and off-line mode (up to 20 Mevps and 33.5 Mev stored in DDR RAM). The sensory integration allows the bio-inspired cochlea limit to provide a concrete range of rpm approaches, which are obtained by the silicon retina.