Reservoir Computing (RC) processes temporal data using a fixed recurrent system and a trained linear readout, making it appealing for hardware-limited neuromorphic settings.Memristive-friendly reservoirs refine this idea by adopting neuron dynamics inspired by resistive devices, but they still rely on dense recurrent matrices that are costly to implement physically.We introduce a Hadamard-based alternative in which the recurrence is replaced by an orthogonal, multiplier-free operator with O(N log N ) complexity and O(N ) parameters.Experiments on timeseries tasks show that the proposed approach matches the performance of dense RC baselines while improving hardware compatibility.

Motivated by advances in physical reservoir computing, we seek models that retain the modularity of echo state networks while enriching their internal dynamics.Recent studies have demonstrated that oscillator networks can achieve this balance, although their simple harmonic nature may limit their expressiveness.Here, we investigate the idea of augmenting harmonic oscillators with non-holonomic (velocitylevel) constraints, known to induce rich, nonlocal behaviors.We implement these constraints intrinsically within each dynamical unit, yielding a model equivalent to the unicycle -the canonical representation of the simplest vehicle.We test the model on three time-series classification benchmarks, achieving competitive or superior accuracy compared to the state of the art, with reservoirs as small as 20 unicycles.

Machine Learning (ML) has become a central force in Artificial Intelligence, driving major breakthroughs in applications that handle increasingly complex data, from images and text sequences to graph structures. While new architectures such as Transformers and Graph Neural Networks continue to redefine performance benchmarks in various domains, these predominantly data-driven methods often neglect critical domain knowledge, practical constraints, and broader contextual factors. This oversight diminishes their trustworthiness and restricts their impact in real-world settings. In this paper, we discuss the need for a more informed approach to ML for complex data. Specifically, we advocate for solutions that explicitly integrate structural awareness to capture underlying relationships in the data, incorporate key technical requirements to ensure safety and compliance with industry standards, embed environmental considerations to promote sustainability and resource efficiency, adhere to established physical principles, and uphold ethical and societal values. By weaving these dimensions together, informed ML can bridge the gap between purely data-centric methods and the nuanced demands of practical applications. We show how this integrated framework not only strengthens model performance but also ensures that ML solutions remain trustworthy, efficient, and sensitive to human ecological, ethical, and regulatory imperatives. Our discussion underscores the transformative potential of Informed ML to drive innovation across diverse domains, setting a new benchmark for responsible and high-impact ML system design.

Reservoir Computing (RC) is a popular approach for modeling dynamical Recurrent Neural Networks, featured by a fixed (i.e., untrained) recurrent reservoir layer. In this paper, we introduce a novel design strategy for deep RC neural networks that is especially suitable to neuromorphic hardware implementations. From the topological perspective, the introduced model presents a multi-level architecture with ring reservoir topology and one-to-one inter-reservoir connections. The proposed design also considers hardware-friendly nonlinearity and noise modeling in the reservoir update equations. We demonstrate the introduced hardware-friendly deep RC architecture in electronic hardware, showing the promising processing capabilities on learning tasks that require both nonlinear computation and short-term memory. Additionally, we validate the effectiveness of the introduced approach on several time-series classification tasks, showing its competitive performance compared to its shallow counterpart, conventional, as well as more recent RC systems. These results emphasize the advantages of the proposed deep architecture for both practical hardware-friendly environments and broader machine learning applications.

A common problem in Message-Passing Neural Networks is oversquashing -- the limited ability to facilitate effective information flow between distant nodes. Oversquashing is attributed to the exponential decay in information transmission as node distances increase. This paper introduces a novel perspective to address oversquashing, leveraging dynamical systems properties of global and local non-dissipativity, that enable the maintenance of a constant information flow rate. We present SWAN, a uniquely parameterized GNN model with antisymmetry both in space and weight domains, as a means to obtain non-dissipativity. Our theoretical analysis asserts that by implementing these properties, SWAN offers an enhanced ability to transmit information over extended distances. Empirical evaluations on synthetic and real-world benchmarks that emphasize long-range interactions validate the theoretical understanding of SWAN, and its ability to mitigate oversquashing.

Time series classification is a pivotal task in modern machine learning, with widespread applications in fields such as healthcare, finance, and cybersecurity. While deep learning methods dominate recent developments, their resource demands and privacy limitations hinder deployment on low-power and decentralized environments. To address these challenges, we introduce Fed2RC, a fully federated and gradient-free approach that integrates the efficiency of Rocket-based feature extraction with the robustness of ridge regression in a privacy-preserving setting. Fed2RC builds upon two key ideas: (i) federated selection and aggregation of high-performing random convolution kernels, and (ii) incremental and communication-efficient updates of ridge classifier parameters using closed-form solutions. Additionally, we propose a novel federated protocol for selecting the global ridge regularization parameter λ, and show how to improve the communication efficiency by matrix factorization techniques. Extensive experiments on the UCR benchmark demonstrate that Fed2RC achieves state-of-the-art results with a fraction of the computation and communication costs. Code to reproduce the experiments can be found at: https://github.com/CasellaJr/Fed2RC.

Recurrent neural networks (RNNs) are computational models regarded as dynamical systems.Modularity is a key ingredient of complex systems.Thus, the composition of RNN modules provides a simple paradigm for building complex computational models, with the potential to approach the human brain capability.We devise strategies for training RNNs assembled into a larger RNN of RNNs, provided with theoretical guarantees of stability that hold during training for the composed global network.Experiments on pixel-by-pixel image classification benchmarks prove the effectiveness of this approach.* This work has been

We propose a novel neural processing unit for artificial neural networks, inspired by the memristive properties of nanowires.Our analysis, framed within the Reservoir Computing paradigm, demonstrates the stability, short-term memory, and fading memory capabilities of the unit.Further experiments on assemblies of nanowire-inspired neurons show promising results in time-series classification tasks.Our introduced approach bridges analog neuromorphic hardware and AI applications, enabling efficient time series processing.

The dynamics of information diffusion within graphs is a critical open issue that heavily influences graph representation learning, especially when considering long-range propagation. This calls for principled approaches that control and regulate the degree of propagation and dissipation of information throughout the neural flow. Motivated by this, we introduce port-Hamiltonian Deep Graph Networks, a novel framework that models neural information flow in graphs by building on the laws of conservation of Hamiltonian dynamical systems. We reconcile under a single theoretical and practical framework both non-dissipative long-range propagation and non-conservative behaviors, introducing tools from mechanical systems to gauge the equilibrium between the two components. Our approach can be applied to general message-passing architectures, and it provides theoretical guarantees on information conservation in time. Empirical results prove the effectiveness of our port-Hamiltonian scheme in pushing simple graph convolutional architectures to state-of-the-art performance in long-range benchmarks.

Graph State Space Models (SSMs) have recently been introduced to enhance Graph Neural Networks (GNNs) in modeling long-range interactions. Despite their success, existing methods either compromise on permutation equivariance or limit their focus to pairwise interactions rather than sequences. Building on the connection between Autoregressive Moving Average (ARMA) and SSM, in this paper, we introduce GRAMA, a Graph Adaptive method based on a learnable ARMA framework that addresses these limitations. By transforming from static to sequential graph data, GRAMA leverages the strengths of the ARMA framework, while preserving permutation equivariance. Moreover, GRAMA incorporates a selective attention mechanism for dynamic learning of ARMA coefficients, enabling efficient and flexible long-range information propagation. We also establish theoretical connections between GRAMA and Selective SSMs, providing insights into its ability to capture long-range dependencies. Experiments on 26 synthetic and real-world datasets demonstrate that GRAMA consistently outperforms backbone models and performs competitively with state-of-the-art methods.

We introduce a novel class of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) paradigm, called Residual Reservoir Memory Networks (ResRMNs). ResRMN combines a linear memory reservoir with a non-linear reservoir, where the latter is based on residual orthogonal connections along the temporal dimension for enhanced long-term propagation of the input. The resulting reservoir state dynamics are studied through the lens of linear stability analysis, and we investigate diverse configurations for the temporal residual connections. The proposed approach is empirically assessed on time-series and pixel-level 1-D classification tasks. Our experimental results highlight the advantages of the proposed approach over other conventional RC models. Code is available at github.com/NennoMP/residualrmn

Graph Neural Networks (GNNs) are models that leverage the graph structure to transmit information between nodes, typically through the message-passing operation. While widely successful, this approach is well-known to suffer from representational collapse as the number of layers increases and insensitivity to the information contained at distant and poorly connected nodes. In this paper, we present a unified view of on the appearance of these issues through the lens of vanishing gradients, using ideas from linear control theory for our analysis. We propose an interpretation of GNNs as recurrent models and empirically demonstrate that a simple state-space formulation of an GNN effectively alleviates these issues at no extra trainable parameter cost. Further, we show theoretically and empirically that (i) Traditional GNNs are by design prone to extreme gradient vanishing even after few layers; (ii) Feature collapse is directly related to the mechanism causing vanishing gradients; (iii) Long-range modeling is most easily achieved by a combination of graph rewiring and vanishing gradient mitigation. We believe our work will help bridge the gap between the recurrent and graph neural network literature and will unlock the design of new deep and performant GNNs.

Echo state networks (ESNs) are time series processing models working under the echo state property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. To bring together the fading memory property and the ability to retain as much memory as possible, in this article, we introduce a new ESN architecture called the Edge of Stability ESN (ES2N). The introduced ES2N model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. In virtue of a thorough mathematical analysis, we prove that the whole eigenspectrum of the Jacobian of the ES2N map can be contained in an annular neighborhood of a complex circle of controllable radius. This property is exploited to tune the ES2N's dynamics close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that ES2N model can reach the theoretical maximum short-term memory capacity (MC). At the same time, in comparison to conventional reservoir approaches, ES2N is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling and real-world time series modeling.

We introduce the Random Oscillators Network (RON), a physically-inspired recurrent model derived from a network of heterogeneous oscillators. Unlike traditional recurrent neural networks, RON keeps the connections between oscillators untrained by leveraging on smart random initialisations, leading to exceptional computational efficiency. A rigorous theoretical analysis finds the necessary and sufficient conditions for the stability of RON, highlighting the natural tendency of RON to lie at the edge of stability, a regime of configurations offering particularly powerful and expressive models. Through an extensive empirical evaluation on several benchmarks, we show four main advantages of RON. 1) RON shows excellent long-term memory and sequence classification ability, outperforming other randomised approaches. 2) RON outperforms fully-trained recurrent models and state-of-the-art randomised models in chaotic time series forecasting. 3) RON provides expressive internal representations even in a small parametrisation regime making it amenable to be deployed on low-powered devices and at the edge. 4) RON is up to two orders of magnitude faster than fully-trained models.

We introduce Reservoir Memory Networks (RMNs), a novel class of Reservoir Computing (RC) models that integrate a linear memory cell with a non-linear reservoir to enhance long-term information retention.We explore various configurations of the memory cell using orthogonal circular shift matrices and Legendre polynomials, alongside nonlinear reservoirs configured as in Echo State Networks and Euler State Networks.Experimental results demonstrate the substantial benefits of RMNs in time-series classification tasks, highlighting their potential for advancing RC applications in areas requiring robust temporal processing.

In the contemporary era of data-driven decision-making, the application of Machine Learning (ML) on complex data (e.g., images, text, sequences, trees, and graphs) has become increasingly pivotal (e.g., Large Language Models and Graph Neural Networks).In this context, there is a gap between purely data-driven models and domain-specific knowledge, requirements, and expertise.In particular, this domain specificity needs to be integrated into the ML models to improve learning generalization, sustainability, trustworthiness, reliability, security, and safety.This additional knowledge can assume different forms, e.g.: software developers require ML to comply with many technical requirements, companies require ML to comply with economic and environmental sustainability, domain experts require ML to be aligned with physical and logical laws, and society requires ML to be aligned with ethical principles.This special session gathers valuable contributions and early findings in the field of Informed ML for Complex Data.Our main objective is to showcase the potential and limitations of new ideas, improvements, or the blending of ML and other research areas in solving real-world problems.

Recurrent Neural Networks are effective for analyzing temporal data, such as time series, but they often require costly and time-intensive training.Echo State Networks simplify the training process by using a fixed recurrent layer, the reservoir, and a trainable output layer, the readout.In sequence classification problems, the readout typically receives only the final state of the reservoir.However, averaging all states can sometimes be beneficial.In this work, we assess whether a weighted average of hidden states can enhance the Echo State Network performance.To this end, we propose a gradient-based, explainable technique to guide the contribution of each hidden state towards the final prediction.We show that our approach outperforms the naive average, as well as other baselines, in time series classification, particularly on noisy data.

The TEACHING project enhances AI applications in pervasive environments via Humanistic Intelligence, fostering synergy between humans and Cyber-Physical Systems of Systems (CPSoS). Here, we present the TEACHING Platform, a microservice-based framework providing the technological advancements to represent humans and CPSoS as containerized software models that interact to mutually empower each other.

Learning Continuous-Time Dynamic Graphs (C-TDGs) requires accurately modeling spatio-temporal information on streams of irregularly sampled events. While many methods have been proposed recently, we find that most message passing-, recurrent- or self-attention-based methods perform poorly on *long-range* tasks. These tasks require correlating information that occurred "far" away from the current event, either spatially (higher-order node information) or along the time dimension (events occurred in the past). To address long-range dependencies, we introduce Continuous-Time Graph Anti-Symmetric Network (CTAN). Grounded within the ordinary differential equations framework, our method is designed for efficient propagation of information. In this paper, we show how CTAN's (i) long-range modeling capabilities are substantiated by theoretical findings and how (ii) its empirical performance on synthetic long-range benchmarks and real-world benchmarks is superior to other methods. Our results motivate CTAN's ability to propagate long-range information in C-TDGs as well as the inclusion of long-range tasks as part of temporal graph models evaluation.

The steelmaking industry could benefit greatly from a reliable technique for predicting the mechanical properties of rolling products. This would lead to significant reductions in time and costs associated with the process.In this paper, we present a novel approach to predict the ultimate tensile strength of hot-rolled steel strips, utilizing the capabilities of recurrent neural models to process temporal data. Our focus is on Reservoir Computing (RC) models, selected for their efficient training characteristics, which are advantageous in production support contexts. In the paper, we introduce two custom RC-based architectures, designed to handle input features distinctively based on their relevance to the steelmaking process. The proposed approach is experimentally validated on a real use case with data originating from a hot rolling steel strip plant. It is compared against standard RC and fully-trainable recurrent neural networks. The results demonstrate the ability of the proposed method to reach a significantly good predictive performance, largely within the threshold set by industry experts. Our custom RC models offer an outstanding balance between predictive performance and computational efficiency, making them highly suitable for this application. In comparison to baseline RC approaches, at substantially the same computational cost, we manage to reduce the prediction error by more than 50%. Moreover, in comparison with fully trained models, we achieve even slightly more accurate predictions while reducing computational cost by more than 20 times. Finally, our results indicate that it is possible to further improve predictive performance through the differentiation of predictive models based on chemical composition similarities.

Federated Echo State Networks proved their efficiency in learning low-resource collaborative settings where data is regulated privacy. In this work, we broaden the applicability of this machine learning approach to a decentralized setting, where we have a set of peers connected through a logical communication topology and cannot rely on a centralized aggregation entity. In particular, we propose Decentralized Incremental Federated Learning (DIncFed), where multiple agents collaborate to learn a readout by leveraging exact consensus strategies. Such strategies include mechanisms for collaboratively aggregating knowledge towards consensus, as well as policies for dynamically updating the communication topology. Experiments prove the efficacy and the efficiency of the proposed learning methodology against a state-of-the-art iterative competitor on multiple benchmarks characterized by different levels of statistical heterogeneity.

Residual connections have been established as a staple for modern deep learning architectures. Most of their applications are cast towards feedforward computing. In this paper, we study the architectural bias of residual connections in the context of recurrent neural networks (RNNs), specifically in the temporal dimension. We frame our discussion from the perspective of Reservoir Computing and dynamical system theory, focusing on important aspects of neural computation like memory capacity, long-term information processing, stability, and nonlinear computation capability. Experiments corroborate the striking advantage brought by temporal residual connections for a plethora of different time series processing tasks, comprehending memory-based, forecasting, and classification problems. • We study residual (skip) connections in the context of RNNs in the temporal dimension. • Temporal residual connections enable long-term processing of time series. • Orthogonal skip connections allow to drive RNNs to the edge-of-stability in an approximate dynamical isometry regime. • Experiments on memory, forecasting, and classification tasks show the benefits of temporal residual connections in RNNs.

In this paper, we investigate neural models based on graph random features for classification tasks. First, we aim to understand when over parameterization, namely generating more features than the ones necessary to interpolate, may be beneficial for the generalization abilities of the resulting models. We employ two measures: one from the algorithmic stability framework and another one based on information theory. We provide empirical evidence from several commonly adopted graph datasets showing that the considered measures, even without considering task labels, can be effective for this purpose. Additionally, we investigate whether these measures can aid in the process of hyperparameters selection. The results of our empirical analysis show that the considered measures have good correlations with the estimated generalization performance of the models with different hyperparameter configurations. Moreover, they can be used to identify good hyperparameters, achieving results comparable to the ones obtained with a classic grid search.

Inspired by the numerical solution of ordinary differential equations, in this paper, we propose a novel Reservoir Computing (RC) model, called the Euler State Network (EuSN). The presented approach makes use of forward Euler discretization and antisymmetric recurrent matrices to design reservoir dynamics that are both stable and non-dissipative by construction. Our mathematical analysis shows that the resulting model is biased towards a unitary effective spectral radius and zero local Lyapunov exponents, intrinsically operating near the edge of stability. Experiments on long-term memory tasks show the clear superiority of the proposed approach over standard RC models in problems requiring effective propagation of input information over multiple time steps. Furthermore, results on time-series classification benchmarks indicate that EuSN can match (or even exceed) the accuracy of trainable Recurrent Neural Networks, while retaining the training efficiency of the RC family, resulting in up to ≈464-fold savings in computation time and ≈1750-fold savings in energy consumption. At the same time, our results on time-series modeling tasks show competitive results against standard RC when the architecture is complemented by direct input-readout connections.

Federated Echo State Networks represent an efficient methodology for learning in pervasive environments with private temporal data due to the low computational cost required by the learning phase.In this paper, we propose Partial Federated Ridge Regression (pFedRR), an approximate, communication-efficient version of the exact method for learning the readout in a federated setting.Each client compresses the local statistics to be exchanged with the server via an importance-based method, which selects the most relevant neurons with respect to the local distribution.We evaluate the methodology on two Human State Monitoring benchmarks, and results show that the importance-based selection of the information significantly reduces the communication cost, while acting as a regularization method to improve the generalization capabilities.

Artificial Intelligence, and in particular Machine Learning, has become ubiquitous in today's society, both revolutionizing and impacting society as a whole.However, it can also lead to algorithmic bias and unfair results, especially when sensitive information is involved.This paper addresses the problem of algorithmic fairness in Machine Learning for temporal data, focusing on ensuring that sensitive time-dependent information does not unfairly influence the outcome of a classifier.In particular, we focus on a class of training-efficient recurrent neural models called Echo State Networks, and show, for the first time, how to leverage local unsupervised adaptation of the internal dynamics in order to build fairer classifiers.Experimental results on real-world problems from physiological sensor data demonstrate the potential of the proposal.

We introduce a novel class of Reservoir Computing (RC) models, a family of efficiently trainable Recurrent Neural Networks based on untrained connections.Aiming to improve the forward propagation of input information through time, we augment standard Echo State Networks (ESNs) with linear reservoir-skip connections modulated by an untrained orthogonal weight matrix.We analyze the mathematical properties of the resulting reservoir systems and show that the dynamical regime of the proposed class of models is controllably close to the edge of stability.Experiments on several time-series classification tasks highlight the striking performance advantage of the proposed approach over standard ESNs.

Features extracted from Deep Neural Networks (DNNs) have proven to be very effective in the context of Content Based Image Retrieval (CBIR). Recently, biologically inspired Hebbian learning algorithms have shown promises for DNN training. In this contribution, we study the performance of such algorithms in the development of feature extractors for CBIR tasks. Specifically, we consider a semi-supervised learning strategy in two steps: first, an unsupervised pre-training stage is performed using Hebbian learning on the image dataset; second, the network is fine-tuned using supervised Stochastic Gradient Descent (SGD) training. For the unsupervised pre-training stage, we explore the nonlinear Hebbian Principal Component Analysis (HPCA) learning rule. For the supervised fine-tuning stage, we assume sample efficiency scenarios, in which the amount of labeled samples is just a small fraction of the whole dataset. Our experimental analysis, conducted on the CIFAR10 and CIFAR100 datasets, shows that, when few labeled samples are available, our Hebbian approach provides relevant improvements compared to various alternative methods.

We introduce a novel Reservoir Computing (RC) approach for multi-dimensional temporal signals.Our proposal is based on routing the different dimensions of the driving input towards different dynamical sub-modules in a multi-reservoir architecture.At the same time, controllable interconnections among the sub-modules allow modeling the interplay between the different dynamics that might be required by the task.Experiments on synthetic and real-world time-series classification problems clearly show the advantages of the proposed approach in dealing with multi-dimensional signals in comparison to standard RC neural networks.

We propose a novel algorithm for performing federated learning with Echo State Networks (ESNs) in a client-server scenario.In particular, our proposal focuses on the adaptation of reservoirs by combining Intrinsic Plasticity with Federated Averaging.The former is a gradientbased method for adapting the reservoir's non-linearity in a local and unsupervised manner, while the latter provides the framework for learning in the federated scenario.We evaluate our approach on real-world datasets from human monitoring, in comparison with the previous approach for federated ESNs existing in literature.Results show that adapting the reservoir with our algorithm provides a significant improvement on the performance of the global model.

Reservoir computing (RC) is a state-of-the-art approach for efficient training in temporal domains.In this paper, we explore new RC architectures that generalise the popular leaky echo state network model (leaky-ESN) introducing an additive orthogonal term outside the nonlinear part of the ESN equation.We investigate the benefits of employing orthogonal matrices in ESNs both inside the nonlinearity and outside of it.We show empirically how to boost the memory capacity towards the theoretical maximum value while still preserving the power of nonlinear computations.Ergo, we optimise the compromise between computing with memory and computing with nonlinearity.The proposed model demonstrates to outperform both leaky-ESN and orthogonal reservoir ESN models on tasks requiring nonlinear computations with memory.* This work has been partially

Continual Learning (CL) on time series data represents a promising but under-studied avenue for real-world applications.We propose two new CL benchmarks for Human State Monitoring.We carefully designed the benchmarks to mirror real-world environments in which new subjects are continuously added.We conducted an empirical evaluation to assess the ability of popular CL strategies to mitigate forgetting in our benchmarks.Our results show that, possibly due to the domainincremental properties of our benchmarks, forgetting can be easily tackled even with a simple finetuning and that existing strategies struggle in accumulating knowledge over a fixed, held-out, test subject.* This work has been partially

The Euler State Network (EuSN) is a recently proposed Reservoir Computing (RC) model where the fixed state dynamics are obtained by discretizing an ordinary differential equation under stability and non-dissipative conditions. As a result, the model is able to effectively propagate input information over time, hugely improving the performance of RC models in tasks requiring long-term memory. Aiming at both reducing the complexity of the reservoir structure and further improving its efficiency, in this paper we propose a minimalistic EuSN architecture where the reservoir is constrained to a fixed bi-directional chain structure. We explore progressive simplifications where the recurrent and the input connections of the reservoir are fully described by a single weight value. While reducing the complexity of the base EuSN, the proposed minimal EuSN approach shows comparable performance on several tasks on time-series classification, thus offering considerable potential advantages, especially in embedded applications and physical implementations.

Graph echo state networks (GESN) are a class of reservoir computing models for the efficient and effective processing of graphs. They compute graph embeddings by the convergence to a fixed point of a dynamical system, randomly initialized according to a generalization of the echo state property, called the graph embedding stability (GES) property. In this paper, we prove new and more accurate bounds for necessary and sufficient GES conditions. Experiments demonstrate how these bounds allow an easier parameter selection and better quality reservoirs.

With the penetration of artificial intelligence (AI) technology into industrial applications, not only computational effectiveness but also computational efficiency in machine learning (ML) methods has been increasingly demanded. Reservoir computing (RC) is an ML framework leveraging a dynamicreservoirfor a nonlinear transformation of sequential inputs and areadoutfor mapping the reservoir state to a desired output. Since only the readout is trained with a simple learning algorithm, RC has attracted much attention as a promising approach to enhance compatibility between high computational performance and low learning cost. In addition, recent studies on physical reservoirs implemented with various physical substrates have boosted the potential of RC in the development of effective and efficient AI hardware. Therefore, it is time to further explore the new frontiers in extremely efficient RC.

Continual Learning (CL) refers to a learning setup where data is non stationary and the model has to learn without forgetting existing knowledge.The study of CL for sequential patterns revolves around trained recurrent networks.In this work, instead, we introduce CL in the context of Echo State Networks (ESNs), where the recurrent component is kept fixed.We provide the first evaluation of catastrophic forgetting in ESNs and we highlight the benefits in using CL strategies which are not applicable to trained recurrent models.Our results confirm the ESN as a promising model for CL and open to its use in streaming scenarios.* This work has been partially supported by the H2020 TEACHING

We draw connections between Reservoir Computing (RC) and Ordinary Differential Equations, introducing a novel class of models called Euler State Networks (EuSNs).The proposed approach is featured by system dynamics that are both stable and non-dissipative, hence enabling an effective transmission of input signals over time.At the same time, EuSN is featured by untrained recurrent dynamics, preserving all the computational advantages of RC models.Through experiments on several benchmarks for time-series classification, we empirically show that EuSN can substantially narrow the performance gap between RC and fully trainable recurrent neural networks.

A critical aspect in Federated Learning is the aggregation strategy for the combination of multiple models, trained on the edge, into a single model that incorporates all the knowledge in the federation. Common Federated Learning approaches for Recurrent Neural Networks (RNNs) do not provide guarantees on the predictive performance of the aggregated model. In this paper we show how the use of Echo State Networks (ESNs), which are efficient state-of-the-art RNN models for time-series processing, enables a form of federation that is optimal in the sense that it produces models mathematically equivalent to the corresponding centralized model. Furthermore, the proposed method is compliant with privacy constraints. The proposed method, which we denote as Incremental Federated Learning, is experimentally evaluated against an averaging strategy on two datasets for human state and activity recognition.

Artificial Recurrent Neural Networks are a powerful information processing abstraction, and Reservoir Computing provides an efficient strategy to build robust implementations by projecting external inputs into high dimensional dynamical system trajectories. In this paper, we propose an extension of the original approach, a local unsupervised learning mechanism we call Phase Transition Adaptation, designed to drive the system dynamics towards the ‘edge of stability’. Here, the complex behavior exhibited by the system elicits an enhancement in its overall computational capacity. We show experimentally that our approach consistently achieves its purpose over several datasets.

We address the efficiency issue for the construction of a deep graph neural network (GNN). The approach exploits the idea of representing each input graph as a fixed point of a dynamical system (implemented through a recurrent neural network), and leverages a deep architectural organization of the recurrent units. Efficiency is gained by many aspects, including the use of small and very sparse networks, where the weights of the recurrent units are left untrained under the stability condition introduced in this work. This can be viewed as a way to study the intrinsic power of the architecture of a deep GNN, and also to provide insights for the set-up of more complex fully-trained models. Through experimental results, we show that even without training of the recurrent connections, the architecture of small deep GNN is surprisingly able to achieve or improve the state-of-the-art performance on a significant set of tasks in the field of graphs classification.

The increasing role of Artificial Intelligence (AI) and Machine Learning (ML) in our lives brought a paradigm shift on how and where the computation is performed. Stringent latency requirements and congested bandwidth moved AI inference from Cloud space towards end-devices. This change required a major simplification of Deep Neural Networks (DNN), with memory-wise libraries or co-processors that perform fast inference with minimal power. Unfortunately, many applications such as natural language processing, time-series analysis and audio interpretation are built on a different type of Artifical Neural Networks (ANN), the so-called Recurrent Neural Networks (RNN), which, due to their intrinsic architecture, remains too complex and heavy to run efficiently on embedded devices. To solve this issue, the Reservoir Computing paradigm proposes sparse untrained non-linear networks, the Reservoir, that can embed temporal relations without some of the hindrances of Recurrent Neural Networks training, and with a lower memory usage. Echo State Networks (ESN) and Liquid State Machines are the most notable examples. In this scenario, we propose a performance comparison of a ESN, designed and trained using Bayesian Optimization techniques, against current RNN solutions. We aim to demonstrate that ESN have comparable performance in terms of accuracy, require minimal training time, and they are more optimized in terms of memory usage and computational efficiency. Preliminary results show that ESN are competitive with RNN on a simple benchmark, and both training and inference time are faster, with a maximum speed-up of 2.35x and 6.60x, respectively.

Machine Learning for graphs is nowadays a research topic of consolidated relevance. Common approaches in the field typically resort to complex deep neural network architectures and demanding training algorithms, highlighting the need for more efficient solutions. The class of Reservoir Computing (RC) models can play an important role in this context, enabling to develop fruitful graph embeddings through untrained recursive architectures. In this paper, we study progressive simplifications to the design strategy of RC neural networks for graphs. Our core proposal is based on shaping the organization of the hidden neurons to follow a ring topology. Experimental results on graph classification tasks indicate that ring-reservoirs architectures enable particularly effective network configurations, showing consistent advantages in terms of predictive performance.

Reservoir Computing (RC) is a well-known strategy for designing Recurrent Neural Networks featured by striking efficiency of training. The crucial aspect of RC is to properly instantiate the hidden recurrent layer that serves as dynamical memory to the system. In this respect, the common recipe is to create a pool of randomly and sparsely connected recurrent neurons. While the aspect of sparsity in the design of RC systems has been debated in the literature, it is nowadays understood mainly as a way to enhance the efficiency of computation, exploiting sparse matrix operations. In this paper, we empirically investigate the role of sparsity in RC network design under the perspective of the richness of the developed temporal representations. We analyze both sparsity in the recurrent connections, and in the connections from the input to the reservoir. Our results point out that sparsity, in particular in input-reservoir connections, has a major role in developing internal temporal representations that have a longer short-term memory of past inputs and a higher dimension.

Gating mechanisms are widely used in the context of Recurrent Neural Networks (RNNs) to improve the network's ability to deal with long-term dependencies within the data. The typical approach for training such networks involves the expensive algorithm of gradient descent and backpropagation. On the other hand, Reservoir Computing (RC) approaches like Echo State Networks (ESNs) are extremely efficient in terms of training time and resources thanks to their use of randomly initialized parameters that do not need to be trained. Unfortunately, basic ESNs are also unable to effectively deal with complex long-term dependencies. In this work, we start investigating the problem of equipping ESNs with gating mechanisms. Under rigorous experimental settings, we compare the behaviour of an ESN with randomized gate parameters (initialized with RC techniques) against several other models, among which a leaky ESN and a fully trained gated RNN. We observe that the use of randomized gates by itself can increase the predictive accuracy of a ESN, but this increase is not meaningful when compared with other techniques. Given these results, we propose a research direction for successfully designing ESN models with gating mechanisms.

The increase in computational power of embedded devices and the latency demands of novel applications brought a paradigm shift on how and where the computation is performed. Although AI inference is slowly moving from the cloud to end-devices with limited resources, time-centric recurrent networks like Long-Short Term Memory remain too complex to be transferred on embedded devices without extreme simplifications and limiting the performance of many notable applications. To solve this issue, the Reservoir Computing paradigm proposes sparse, untrained non-linear networks, the Reservoir, that can embed temporal relations without some of the hindrances of Recurrent Neural Networks training, and with a lower memory occupation. Echo State Networks (ESN) and Liquid State Machines are the most notable examples. In this scenario, we propose EchoBay , a comprehensive C++ library for ESN design and training. EchoBay is architecture-agnostic to guarantee maximum performance on different devices (whether embedded or not), and it offers the possibility to optimize and tailor an ESN on a particular case study, reducing at the minimum the effort required on the user side. This can be done thanks to the Bayesian Optimization (BO) process, which efficiently and automatically searches hyper-parameters that maximize a fitness function. Additionally, we designed different optimization techniques that take in consideration resource constraints of the device to minimize memory footprint and inference time. Our results in different scenarios show an average speed-up in training time of 119x compared to Grid and Random search of hyper-parameters, a decrease of 94% of trained models size and 95% in inference time, maintaining comparable performance for the given task. The EchoBay library is Open Source and publicly available at https://github.com/necst/Echobay.

Abstract Smart robotic environments combine traditional (ambient) sensing devices and mobile robots. This combination extends the type of applications that can be considered, reduces their complexity, and enhances the individual values of the devices involved by enabling new services that cannot be performed by a single device. To reduce the amount of preparation and preprogramming required for their deployment in real‐world applications, it is important to make these systems self‐adapting. The solution presented in this paper is based upon a type of compositional adaptation where (possibly multiple) plans of actions are created through planning and involve the activation of pre‐existing capabilities. All the devices in the smart environment participate in a pervasive learning infrastructure, which is exploited to recognize which plans of actions are most suited to the current situation. The system is evaluated in experiments run in a real domestic environment, showing its ability to proactively and smoothly adapt to subtle changes in the environment and in the habits and preferences of their user(s), in presence of appropriately defined performance measuring functions.

Metric learning has the aim to improve classification accuracy by learning a distance measure which brings data points from the same class closer together and pushes data points from different classes further apart. Recent research has demonstrated that metric learning approaches can also be applied to trees, such as molecular structures, abstract syntax trees of computer programs, or syntax trees of natural language, by learning the cost function of an edit distance, i.e. the costs of replacing, deleting, or inserting nodes in a tree. However, learning such costs directly may yield an edit distance which violates metric axioms, is challenging to interpret, and may not generalize well. In this contribution, we propose a novel metric learning approach for trees which we call embedding edit distance learning (BEDL) and which learns an edit distance indirectly by embedding the tree nodes as vectors, such that the Euclidean distance between those vectors supports class discrimination. We learn such embeddings by reducing the distance to prototypical trees from the same class and increasing the distance to prototypical trees from different classes. In our experiments, we show that BEDL improves upon the state-of-the-art in metric learning for trees on six benchmark data sets, ranging from computer science over biomedical data to a natural-language processing data set containing over 300,000 nodes.

The extension of Recurrent Neural Networks (RNNs) in the direction of deep learning is a topic that is gaining more and more attention in the neural networks community. The study of deep RNNs opened a number of intriguing research questions on the actual role played by layering in the architectural design of RNNs. Recently, the introduction of the Deep Echo State Network (DeepESN) model allowed to start addressing such open issues in literature, contributing to shed light on the intrinsic properties of state dynamics developed by hierarchical compositions of recurrent layers. This contribution intends to present a unified view over the major advancements in the study of DeepESNs, enabling to directly point out the natural advantages of a layered construction of recurrent networks for temporal data processing.

This work proposes a first study, through empirical assessment, of a deep recursive Neural Network (RecNN) architecture for tree structured data exploiting the efficient design of the Echo State Network (ESN) framework. Three benchmark tasks for trees allow us to assess the potentiality of the novel Deep Tree ESN (DeepTESN) model with respect to the shallow counterpart (Tree ESN) and literature results (including hidden tree Markov models and kernel based approaches) in different conditions and according to both efficiency and predictive performance.

The detection of socialization events is useful to build indicators about social isolation of people, which is an important indicator in e-health applications. On the other hand, it is rather difficult to achieve with non-invasive solutions. This paper reports about the currently work-in-progress on the technological solution for the detection of socialization events adopted in the DOREMI project.

In this paper we introduce the Graph Echo State Network (GraphESN) model, a generalization of the Echo State Network (ESN) approach to graph domains. GraphESNs allow for an efficient approach to Recursive Neural Networks (RecNNs) modeling extended to deal with cyclic/acyclic, directed/undirected, labeled graphs. The recurrent reservoir of the network computes a fixed contractive encoding function over graphs and is left untrained after initialization, while a feed-forward readout implements an adaptive linear output function. Contractivity of the state transition function implies a Markovian characterization of state dynamics and stability of the state computation in presence of cycles. Due to the use of fixed (untrained) encoding, the model represents both an extremely efficient version and a baseline for the performance of recursive models with trained connections. The performance are shown on standard benchmark tasks from Chemical domains, allowing the comparison with both Neural Network and Kernel-based approaches for graphs.