Reservoir Computing (RC) processes temporal data using a fixed recurrent system and a trained linear readout, making it appealing for hardware-limited neuromorphic settings.Memristive-friendly reservoirs refine this idea by adopting neuron dynamics inspired by resistive devices, but they still rely on dense recurrent matrices that are costly to implement physically.We introduce a Hadamard-based alternative in which the recurrence is replaced by an orthogonal, multiplier-free operator with O(N log N ) complexity and O(N ) parameters.Experiments on timeseries tasks show that the proposed approach matches the performance of dense RC baselines while improving hardware compatibility.

Motivated by advances in physical reservoir computing, we seek models that retain the modularity of echo state networks while enriching their internal dynamics.Recent studies have demonstrated that oscillator networks can achieve this balance, although their simple harmonic nature may limit their expressiveness.Here, we investigate the idea of augmenting harmonic oscillators with non-holonomic (velocitylevel) constraints, known to induce rich, nonlocal behaviors.We implement these constraints intrinsically within each dynamical unit, yielding a model equivalent to the unicycle -the canonical representation of the simplest vehicle.We test the model on three time-series classification benchmarks, achieving competitive or superior accuracy compared to the state of the art, with reservoirs as small as 20 unicycles.

Recurrent neural networks (RNNs) are computational models regarded as dynamical systems.Modularity is a key ingredient of complex systems.Thus, the composition of RNN modules provides a simple paradigm for building complex computational models, with the potential to approach the human brain capability.We devise strategies for training RNNs assembled into a larger RNN of RNNs, provided with theoretical guarantees of stability that hold during training for the composed global network.Experiments on pixel-by-pixel image classification benchmarks prove the effectiveness of this approach.* This work has been

We propose a novel neural processing unit for artificial neural networks, inspired by the memristive properties of nanowires.Our analysis, framed within the Reservoir Computing paradigm, demonstrates the stability, short-term memory, and fading memory capabilities of the unit.Further experiments on assemblies of nanowire-inspired neurons show promising results in time-series classification tasks.Our introduced approach bridges analog neuromorphic hardware and AI applications, enabling efficient time series processing.

Graph State Space Models (SSMs) have recently been introduced to enhance Graph Neural Networks (GNNs) in modeling long-range interactions. Despite their success, existing methods either compromise on permutation equivariance or limit their focus to pairwise interactions rather than sequences. Building on the connection between Autoregressive Moving Average (ARMA) and SSM, in this paper, we introduce GRAMA, a Graph Adaptive method based on a learnable ARMA framework that addresses these limitations. By transforming from static to sequential graph data, GRAMA leverages the strengths of the ARMA framework, while preserving permutation equivariance. Moreover, GRAMA incorporates a selective attention mechanism for dynamic learning of ARMA coefficients, enabling efficient and flexible long-range information propagation. We also establish theoretical connections between GRAMA and Selective SSMs, providing insights into its ability to capture long-range dependencies. Experiments on 26 synthetic and real-world datasets demonstrate that GRAMA consistently outperforms backbone models and performs competitively with state-of-the-art methods.

We introduce a novel class of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) paradigm, called Residual Reservoir Memory Networks (ResRMNs). ResRMN combines a linear memory reservoir with a non-linear reservoir, where the latter is based on residual orthogonal connections along the temporal dimension for enhanced long-term propagation of the input. The resulting reservoir state dynamics are studied through the lens of linear stability analysis, and we investigate diverse configurations for the temporal residual connections. The proposed approach is empirically assessed on time-series and pixel-level 1-D classification tasks. Our experimental results highlight the advantages of the proposed approach over other conventional RC models. Code is available at github.com/NennoMP/residualrmn

Since the recognition in the early 1990s of the vanishing/exploding (V/E) gradient issue plaguing the training of neural networks (NNs), significant efforts have been exerted to overcome this obstacle. However, a clear solution to the V/E issue remained elusive so far. The pursuit of approximate dynamical isometry, i.e., parameter configurations where the singular values of the input-output Jacobian (IOJ) are tightly distributed around 1, leads to the derivation of an NN's architecture that shares common traits with the popular residual network (ResNet) model. Instead of skipping connections between layers, the idea is to filter the previous activations orthogonally and add them to the nonlinear activations of the next layer, realizing a convex combination between them. Remarkably, the impossibility of the gradient updates to either vanish or explode is demonstrated with analytical bounds that hold even in the infinite depth case. The effectiveness of this method is empirically proved by means of training via backpropagation an extremely deep multilayer perceptron (MLP) of 50k layers, and an Elman NN to learn long-term dependencies in the input of 10k time steps in the past. Compared with other architectures specifically devised to deal with the V/E problem, e.g., LSTMs, the proposed model is way simpler yet more effective. Surprisingly, a single-layer vanilla recurrent NN (RNN) can be enhanced to reach state-of-the-art performance, while converging super fast; for instance, on the psMNIST task, it is possible to get test accuracy of over 94% in the first epoch, and over 98% after just ten epochs.

Echo state networks (ESNs) are time series processing models working under the echo state property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. To bring together the fading memory property and the ability to retain as much memory as possible, in this article, we introduce a new ESN architecture called the Edge of Stability ESN (ES2N). The introduced ES2N model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. In virtue of a thorough mathematical analysis, we prove that the whole eigenspectrum of the Jacobian of the ES2N map can be contained in an annular neighborhood of a complex circle of controllable radius. This property is exploited to tune the ES2N's dynamics close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that ES2N model can reach the theoretical maximum short-term memory capacity (MC). At the same time, in comparison to conventional reservoir approaches, ES2N is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling and real-world time series modeling.

We introduce the Random Oscillators Network (RON), a physically-inspired recurrent model derived from a network of heterogeneous oscillators. Unlike traditional recurrent neural networks, RON keeps the connections between oscillators untrained by leveraging on smart random initialisations, leading to exceptional computational efficiency. A rigorous theoretical analysis finds the necessary and sufficient conditions for the stability of RON, highlighting the natural tendency of RON to lie at the edge of stability, a regime of configurations offering particularly powerful and expressive models. Through an extensive empirical evaluation on several benchmarks, we show four main advantages of RON. 1) RON shows excellent long-term memory and sequence classification ability, outperforming other randomised approaches. 2) RON outperforms fully-trained recurrent models and state-of-the-art randomised models in chaotic time series forecasting. 3) RON provides expressive internal representations even in a small parametrisation regime making it amenable to be deployed on low-powered devices and at the edge. 4) RON is up to two orders of magnitude faster than fully-trained models.

We introduce Reservoir Memory Networks (RMNs), a novel class of Reservoir Computing (RC) models that integrate a linear memory cell with a non-linear reservoir to enhance long-term information retention.We explore various configurations of the memory cell using orthogonal circular shift matrices and Legendre polynomials, alongside nonlinear reservoirs configured as in Echo State Networks and Euler State Networks.Experimental results demonstrate the substantial benefits of RMNs in time-series classification tasks, highlighting their potential for advancing RC applications in areas requiring robust temporal processing.

Recurrent Neural Networks are effective for analyzing temporal data, such as time series, but they often require costly and time-intensive training.Echo State Networks simplify the training process by using a fixed recurrent layer, the reservoir, and a trainable output layer, the readout.In sequence classification problems, the readout typically receives only the final state of the reservoir.However, averaging all states can sometimes be beneficial.In this work, we assess whether a weighted average of hidden states can enhance the Echo State Network performance.To this end, we propose a gradient-based, explainable technique to guide the contribution of each hidden state towards the final prediction.We show that our approach outperforms the naive average, as well as other baselines, in time series classification, particularly on noisy data.

Residual connections have been established as a staple for modern deep learning architectures. Most of their applications are cast towards feedforward computing. In this paper, we study the architectural bias of residual connections in the context of recurrent neural networks (RNNs), specifically in the temporal dimension. We frame our discussion from the perspective of Reservoir Computing and dynamical system theory, focusing on important aspects of neural computation like memory capacity, long-term information processing, stability, and nonlinear computation capability. Experiments corroborate the striking advantage brought by temporal residual connections for a plethora of different time series processing tasks, comprehending memory-based, forecasting, and classification problems. • We study residual (skip) connections in the context of RNNs in the temporal dimension. • Temporal residual connections enable long-term processing of time series. • Orthogonal skip connections allow to drive RNNs to the edge-of-stability in an approximate dynamical isometry regime. • Experiments on memory, forecasting, and classification tasks show the benefits of temporal residual connections in RNNs.

Artificial Intelligence, and in particular Machine Learning, has become ubiquitous in today's society, both revolutionizing and impacting society as a whole.However, it can also lead to algorithmic bias and unfair results, especially when sensitive information is involved.This paper addresses the problem of algorithmic fairness in Machine Learning for temporal data, focusing on ensuring that sensitive time-dependent information does not unfairly influence the outcome of a classifier.In particular, we focus on a class of training-efficient recurrent neural models called Echo State Networks, and show, for the first time, how to leverage local unsupervised adaptation of the internal dynamics in order to build fairer classifiers.Experimental results on real-world problems from physiological sensor data demonstrate the potential of the proposal.

We introduce a novel class of Reservoir Computing (RC) models, a family of efficiently trainable Recurrent Neural Networks based on untrained connections.Aiming to improve the forward propagation of input information through time, we augment standard Echo State Networks (ESNs) with linear reservoir-skip connections modulated by an untrained orthogonal weight matrix.We analyze the mathematical properties of the resulting reservoir systems and show that the dynamical regime of the proposed class of models is controllably close to the edge of stability.Experiments on several time-series classification tasks highlight the striking performance advantage of the proposed approach over standard ESNs.

Reservoir computing (RC) is a state-of-the-art approach for efficient training in temporal domains.In this paper, we explore new RC architectures that generalise the popular leaky echo state network model (leaky-ESN) introducing an additive orthogonal term outside the nonlinear part of the ESN equation.We investigate the benefits of employing orthogonal matrices in ESNs both inside the nonlinearity and outside of it.We show empirically how to boost the memory capacity towards the theoretical maximum value while still preserving the power of nonlinear computations.Ergo, we optimise the compromise between computing with memory and computing with nonlinearity.The proposed model demonstrates to outperform both leaky-ESN and orthogonal reservoir ESN models on tasks requiring nonlinear computations with memory.* This work has been partially