The use of hyperspectral imaging (HSI) for autonomous navigation is a promising field of research that aims at improving the accuracy and robustness of detection, tracking, and scene understanding systems based on vision sensors. The combination of advanced computer algorithms, such as deep neural networks (DNNs), and small-size snapshot HSI cameras allows to strengthen the reliability of those vision systems. Using HSI, some intrinsic limitations of greyscale and RGB imaging in depicting physical properties of targets related to the spectral reflectance of materials (metamerism) are overcome. Despite the promising results of many published HSI-based computer vision developments, the strict requirements of safety-critical applications such as autonomous driving systems (ADS) regarding latency, resource consumption, and security are prompting the migration of machine learning (ML)-based solutions to edge platforms. This involves a thorough software/hardware co-design scheme to distribute and optimize the tasks efficiently among the limited resources of computing platforms. With respect to inference, the over-parameterized nature of DNNs poses significant computational challenges for real-time on-the-edge deployment. In addition, the intensive data preprocessing required by HSI, which is frequently overlooked, must be carefully managed in terms of memory arrangement and inter-task communication to enable an efficient integrated pipeline design on a system on chip (SoC). This work presents a set of optimization techniques for the practical co-design of a DNN-based HSI segmentation processor deployed on a field programmable gate array (FPGA)-based SoC targeted at ADS, including key optimizations such as functional software/hardware task distribution, hardware-aware preprocessing, ML model compression, and a complete pipelined deployment. Applied compression techniques significantly reduce the complexity of the designed DNN to 24.34% of the original operations and to 1.02% of the original number of parameters, achieving a 2.86× speed-up in the inference task without noticeable degradation of the segmentation accuracy.

As the deployment of artificial intelligence (AI) algorithms at edge devices becomes increasingly prevalent, enhancing the robustness and reliability of autonomous AI-based perception and decision systems is becoming as relevant as precision and performance, especially in applications areas considered safety-critical such as autonomous driving and aerospace. This paper delves into the robustness assessment in embedded Deep Neural Networks (DNNs), particularly focusing on the impact of parameter perturbations produced by single event upsets (SEUs) on convolutional neural networks (CNN) for image semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit sensitivity of various encoder–decoder models to soft errors, this study thoroughly investigates the vulnerability of segmentation DNNs to SEUs and evaluates the consequences of techniques like model pruning and parameter quantization on the robustness of compressed models aimed at embedded implementations. The findings offer valuable insights into the mechanisms underlying SEU-induced failures that allow for evaluating the robustness of DNNs once trained in advance. Moreover, based on the collected data, we propose a set of practical lightweight error mitigation techniques with no memory or computational cost suitable for resource-constrained deployments. The code used to perform the fault injection (FI) campaign is available at https://github.com/jonGuti13/TensorFI2, while the code to implement proposed techniques is available at https://github.com/jonGuti13/parameterProtection.

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

Driver Assistance Systems (DAS) have been progressively incorporated into commercial vehicles in recent years. All these systems are paving the way for the forthcoming autonomous vehicle which will become a reality in the near future. Existing systems are based on numerous electronic systems with advanced skills, high performances, and high degrees of adaptability and intelligence. As is to be expected, these cutting-edge features require, in most cases, the use of powerful computing platforms. However, the deployment of such platforms is not an easy task, since they have to be integrated in the vehicle where there exist important restrictions regarding size, power consumption and cost. In this sense, every smart proposal aimed at reducing the complexity of these systems without degrading performance, is always a valuable contribution in the field. In this work, we propose a methodology to reduce the dimensionality of a driver distraction recognition system. The methodology is based on a multi-objective genetic algorithm that looks for the minimum set of useful features collected during the driving task and also for the simplest recognition system. The recognition algorithm is an Extreme Learning Machine (ELM) whose simplicity and fast learning procedure make it especially suitable to be used by a Genetic Algorithm which needs to evaluate thousands of candidate solutions. The proposed methodology has been tested with a real-world database collected from different drivers performing an itinerary with an instrumented car. The results obtained validate the proposal as a method to reduce the complexity of a driver distraction recognition system.

We present a structured dataset for the research and development of automated driving systems (ADS) supported by hyperspectral imaging (HSI). The dataset contains per-pixel manually annotated images selected from videos recorded in real driving conditions that have been organized according to four environment parameters: season, daytime, road type, and weather conditions. The aim is to provide high data diversity and facilitate the automatic generation of data subsets for the evaluation of machine learning (ML) techniques applied to the research of ADS in different driving scenarios and environmental conditions. The video sequences have been captured with a small-size 25-band VNIR (Visible-NearlnfraRed) snapshot hyperspectral camera mounted on a driving automobile. The current selection of classes for image annotation is aimed to provide reliable data for the spectral analysis of the items in the scenes; it is thus based on material surface reflectance patterns (spectral signatures). It is foreseen that future versions of the dataset will also incorporate alternative dense semantic labeling of the annotated images. The first version of the dataset, named HSI-Drive v1.0, is publicly available for download33http://ipaccess.ehu.eus/HSI-Drive.

Automotive ride comfort has become an important research topic in recent years due to the increasing level of automation in currently produced cars. These premises also apply to manned cars. In this work, a hybrid hardware/software extreme learning machine for improved ride comfort in automobiles is proposed. This system is based on a single-chip implementation able to provide real-time information about the level of ride comfort by classifying driving data into several comfort classes. To develop this system, unsupervised hierarchical clustering analysis (HCA) and supervised extreme learning machine (ELM) have been used jointly, to enhance the overall performance of the entire system, reaching classification success rates of up to 95%. This approach has been implemented on a Xilinx Zynq-7000 programmable system-on-chip. This chip is able to process data in real time and to identify the comfort class, achieving low latency marks and high operational frequencies due to its DSP-based implementation. These performance and accuracy marks, together with its low power consumption make this development suitable for novel practical implementations in current production cars.

Autonomy, adaptability and reactivity are key capabilities of intelligent agents. Many applications of intelligent agents, such as control of ubiquitous computing environments or autonomous robotic systems, demand not only high performance and modeling capability but also the appropriate device or architecture (hardware and related software) for implementing the agent in a real environment. To deal with these challenges a new algorithm, termed PWM-FIS ELM, is proposed. It combines a particular type of adaptive fuzzy inference system with piecewise multi-linear behaviour (PWM-FIS), and an extreme learning machine (ELM) training scheme. The PWM-FIS is suitable for the development of efficient high-performance System-on-Chip (SoC), while ELM provides fast training, good generalization ability, and universal approximation capability. The proposed algorithm outperforms previous results obtained by the authors using the PWM-FIS endowed with a conventional two-pass training algorithm (i.e., least square estimator plus back propagation gradient descent method). Experimental results obtained in an inhabited intelligent environment are provided.

Hyperspectral remote sensing is becoming an active research field in the last decades thanks to the availability of efficient machine learning algorithms and also to the ever-increasing computation power. However, there exist application domains (e.g., embedded applications) in which the deployment of this kind of systems becomes unfeasible due to the high requirements related to the size, power consumption or processing speed. A way to overcome this trouble consists on using any method able to scale-down the dimensionality of the problem and/or to reduce the complexity of the machine learning models. In this paper, we propose the use of a multiobjective genetic algorithm to minimize both the dimension of the input space and the size of the machine learning model. In particular, we have developed a hyperspectral image classifier based on an Extreme Learning Machine (ELM) for which the number of system inputs (dimensionality) and the number of hidden neurons are minimized without decreasing its performance. The system is evaluated by using a known benchmark dataset.

Advanced Driving Assitance Systems (ADAS) cover a wide range of systems that aim to provide increasingly a safe and efficient driving. Many of these systems are endowed with some intelligent skills which are, in many cases, addressed by means of Soft Computing (SC) paradigms like Neural Networks (NN) or fuzzy systems among others. However, SC algorithms require normally large computational resources which are incompatible with the kind of the electronic systems that can be deployed in cars where the size, cost and power consumption are always very restrictive. In this paper we present a NN-based driving recognition system, able to model the driving style of different drivers. Such a system could be used to detect abnormal driving behaviours and hence, to avoid dangerous situations. In order to obtain an optimized network in terms of size and complexity, we make the design by using a multiobjective genetic algorithm that provides at the same time a reduced number of input variables and a low number of neurons in the network. To make feasible the operation of the algorithm, we select Extreme Learning Machines as NNs as they have a learning mechanism very fast and precise.

This paper presents a methodology to implement large Neural Networks based classifiers in low-cost FPGAs. The idea is to divide the large Neural Network into several smaller networks which can easily be implemented in small devices. Then, a Multiple Classifier Ensemble is used to joint the results of each small network and thus provide the output of the system. To validate the proposal a classification experiment of terrain images of satellite has been developed and implemented. Obtained results related the size, velocity and performance of the implemented system confirm the viability of the methodology.

The availability of advanced driver assistance systems (ADAS), for safety and well-being, is becoming increasingly important for avoiding traffic accidents caused by fatigue, stress, or distractions. For this reason, automatic identification of a driver from among a group of various drivers (i.e. real-time driver identification) is a key factor in the development of ADAS, mainly when the driver's comfort and security is also to be taken into account. The main focus of this work is the development of embedded electronic systems for in-vehicle deployment of driver identification models. We developed a hybrid model based on artificial neural networks (ANN), and cepstral feature extraction techniques, able to recognize the driving style of different drivers. Results obtained show that the system is able to perform real-time driver identification using non-intrusive driving behavior signals such as brake pedal signals and gas pedal signals. The identification of a driver from within groups with a reduced number of drivers yields promising identification rates (e.g. 3-driver group yield 84.6%). However, real-time development of ADAS requires very fast electronic systems. To this end, an FPGA-based hardware coprocessor for acceleration of the neural classifier has been developed. The coprocessor core is able to compute the whole ANN in less than 4 μs.

The powerful synergy of neural networks and reconfigurable hardware provides a solid foundation for the development of high performance embedded systems able to efficiently adapt to changing requirements. Adaptation at different levels - ranging from the physical level to the system level-can be combined to develop efficient solutions by means of FPGA technology. In this work, a multilevel adaptation scheme for the development of intelligent agents is proposed. Software learning algorithms are applied to adapt the agent behavior (i.e. neural network parameters) at the system level, while dynamic partial reconfiguration (DPR) is used to modify the agent at the physical and architectural level (i.e. neural network topology). Firstly, a multilevel adaptive intelligent agent is able to manage its resources efficiently in order to meet time-varying demands such as speed performance and power consumption. Secondly, from the behavioral viewpoint, multilevel adaptation provides the intelligent agent with high plasticity and flexibility. An FPGA-based intelligent agent has been successfully deployed for a real-time control problem in an inhabited intelligent environment. Results obtained show that the agent is able to adapt itself to changes in the environment in a lifelong mode.

This paper presents the development of an embedded intelligent agent able to perform real-time control of ambient-intelligence environments. The system has been implemented as a system-on-programmable chip (SoPC) on a field programmable gate array (FPGA). The scheme used for realizing the intelligent agent is an adaptive neuro-fuzzy system (NFS) enhanced with a principal component analysis (PCA) pre-processor. The PCA pre-processing stage allows a reduction of the input dimensions (features) with no meaningful loss of modeling capability. As a consequence, the computational complexity of the system is significantly reduced, allowing its implementation on a single electronic device. The NFS-PCA agent has been tested with data obtained in a real ubiquitous computing environment test bed. Results obtained show that the agent is able to perform real-time control of the environment in a proactive and non-intrusive way, and also to adapt to changes of user's preferences in a life-long mode.

In this paper we propose to apply the Dynamic Partial Reconfiguration (DPR) technology to embedded systems intended for Intelligent Environments. To reach this goal, we have developed a system based on a Field Programmable Gate Array (FPGA) in which high performance hardware modules can be reconfigured on-line according to the necessities of the system at each moment. Two different implementations have been carried out to measure the time required to reconfigure each module and also to measure the FPGA resources that can be saved if we keep configured only the modules that are required at each time. The Obtained results show how this technique offers advantages in cost, size and power when applied to embedded systems for intelligent environments.

This paper presents the development of a neuro-fuzzy agent for ambient-intelligence environments. The agent has been implemented as a system-on-chip (SoC) on a reconfigurable device, i.e., a field-programmable gate array. It is a hardware/software (HW/SW) architecture developed around a MicroBlaze processor (SW partition) and a set of parallel intellectual property cores for neuro-fuzzy modeling (HW partition). The SoC is an autonomous electronic device able to perform real-time control of the environment in a personalized and adaptive way, anticipating the desires and needs of its inhabitants. The scheme used to model the intelligent agent is a particular class of an adaptive neuro-fuzzy inference system with piecewise multilinear behavior. The main characteristics of our model are computational efficiency, scalability, and universal approximation capability. Several online experiments have been performed with data obtained in a real ubiquitous computing environment test bed. Results obtained show that the SoC is able to provide high-performance control and adaptation in a life-long mode while retaining the modeling capabilities of similar agent-based approaches implemented on larger computing machines.

This paper describes the development of efficient hardware/software (HW/SW) neuro-fuzzy systems. The model used in this work consists of an adaptive neuro-fuzzy inference system modified for efficient HW/SW implementation. The design of two different on-chip approaches are presented: a high-performance parallel architecture for offline training and a pipelined architecture suitable for online parameter adaptation. Details of important aspects concerning the design of HW/SW solutions are given. The proposed architectures have been implemented using a system-on-a-programmable-chip. The device contains an embedded-processor core and a large field programmable gate array (FPGA). The processor provides flexibility and high precision to implement the learning algorithms, while the FPGA allows the development of high-speed inference architectures for real-time embedded applications.

Presents a fuzzy method for the recognition of strings of fuzzy symbols containing substitution, deletion, and insertion errors. As a preliminary step, we propose a fuzzy automaton to calculate a similarity value between strings. The adequate selection of fuzzy operations for computing the transitions of the fuzzy automaton allows us to obtain different string similarity definitions (including the Levenshtein distance). A deformed fuzzy automaton based on this fuzzy automaton is then introduced in order to handle strings of fuzzy symbols. The deformed fuzzy automaton enables the classification of such strings having an undetermined number of insertion, deletion and substitution errors. The selection of the parameters determining the deformed fuzzy automaton behavior would allow to implement recognizers adapted to different problems. The paper also presents algorithms that implement the deformed fuzzy automaton. Experimental results show good performance in correcting these kinds of errors.

This paper introduces the use of digital signal processors DSPs to implement intelligent control ICT algorithms. We show that DSPs, which are CISC microprocessors designed for real-time processing of digital signals, are also suitable for the implementation of ICT algorithms, such as neural networks and fuzzy systems. These systems are very close to signal processing applications in many aspects. For example, both involve multiply and multiply-accumulate operations, and both require an intensive data treatment and efficient I O peripherals. A detailed description of the software implementation of a feedforward neural network and a fuzzy inference algorithm is included. The performance achieved in both implementations is provided. The main advantages of the proposed alternative are low cost, complete flexibility in the selection of algorithms, and high processing speed in comparison with a standard microprocessor or microcontroller embedded system.