Reservoir Computing (RC) is a time-efficient computational paradigm derived from Recurrent Neural Networks (RNNs). The Simple Cycle Reservoir (SCR) is an RC model that stands out for its minimalistic design, offering extremely low construction complexity and proven capability of universally approximating time-invariant causal fading memory filters, even in the linear dynamics regime. This paper introduces Multiple Simple Cycle Reservoirs (MSCRs), a multi-reservoir framework that extends Echo State Networks (ESNs) by replacing a single large reservoir with multiple interconnected SCRs. We demonstrate that MSCRs optimized with Particle Swarm Optimization (MSCR-PSO) outperform existing multi-reservoir models, achieving competitive predictive performance with a lower-dimensional state space. By modeling interconnections as a weighted Directed Acyclic Graph (DAG), our approach enables flexible, task-specific network topology adaptation. Numerical simulations on three benchmark time-series prediction tasks confirm these advantages over rival algorithms. These findings highlight the potential of MSCR-PSO as a promising framework for optimizing multi-reservoir systems, providing a foundation for further advancements and applications of interconnected SCRs for developing efficient AI devices.

With a growing data privacy concern, federated learning has emerged as a promising framework to train machine learning models without sharing locally distributed data. In federated learning, local model training by multiple clients and model integration by a server are repeated only through model parameter sharing. Most existing federated learning methods assume training deep learning models, which are often computationally demanding. To deal with this issue, we propose federated learning methods with reservoir state analysis to seek computational efficiency and data privacy protection simultaneously. Specifically, our method relies on Mahalanobis Distance of Reservoir States (MD-RS) method targeting time series anomaly detection, which learns a distribution of reservoir states for normal inputs and detects anomalies based on a deviation from the learned distribution. Iterative updating of statistical parameters in the MD-RS enables incremental federated learning (IncFed MD-RS). We evaluate the performance of IncFed MD-RS using benchmark datasets for time series anomaly detection. The results show that IncFed MD-RS outperforms other federated learning methods with deep learning and reservoir computing models particularly when clients’ data are relatively short and heterogeneous. We demonstrate that IncFed MD-RS is robust against reduced sample data compared to other methods. We also show that the computational cost of IncFed MD-RS can be reduced by subsampling from the reservoir states without performance degradation. The proposed method is beneficial especially in anomaly detection applications where computational efficiency, algorithm simplicity, and low communication cost are required.

Online classification of multivariate time series is crucial across diverse domains, from healthcare monitoring to industrial systems. Reservoir computing (RC), which employs a fixed, randomly initialized recurrent neural network to encode temporal data sequentially, has emerged as a promising approach due to its efficient training and ability to capture complex temporal patterns. However, standard RC implementations rely on regression-based readouts that produce temporally unstable outputs and limit interpretability. We introduce Gaussian Reservoir State Analysis (GRSA), a novel approach that combines reservoirs’ temporal processing capabilities with distribution-based classification. GRSA models reservoir responses to sequences in each class using multivariate Gaussian distribution and performs classification based on statistical distances. Through comprehensive evaluation using the UEA Multivariate Time Series Classification Archive, we demonstrate that GRSA significantly outperforms the regression-based RC while providing more stable and interpretable outputs. The method enables continuous classification by generating predictions at each time step and maintains robust performance even with shortened test sequences, making it particularly suitable for early classification tasks. GRSA’s architectural simplicity allows for various extensions and adaptations across different application domains.

In Reservoir Computing methods, Multiple Reservoir Echo State Networks (MRESNs) with multiple reservoir encoders can have better computational abilities than the standard single Echo State Network on many time-series processing tasks. However, several hand-crafted network topologies have been widely adopted for building most existing MRESN-based models, which limits the potential of their performance of an MRESN on a given temporal processing task. In this work, we propose a new method to improve the computational ability of an MRESN based on the perspective of the neural architecture search. To accomplish this goal, we use a binary-encoding method to represent the reservoir network topology of an MRESN and develop a Genetic Algorithm (GA)-based approach to seek a desirable network topology. Experimental results show that MRESNs with the corresponding optimal network topologies searched by our proposed approach have better computational abilities than the baseline MRESN models on four benchmark time-series processing datasets.

This paper proposes a physical-device-based time-domain fading channel prediction scheme using spin-wave reservoir computing. We numerically construct a spin-wave reservoir chip that adopts new spin-wave transducers named Film-penetrating transducers (FPTs). We arrange three FPTs as exciters and forty-nine FPTs as detectors connected to the reservoir input and readout, respectively. We feed communication channel information collected in an actual fading environment to evaluate the prediction performance. We calculate the average root mean squared errors (RMSEs) and obtain the symbol error rates (SERs) for various forecasting time lengths. We find that our proposed scheme can achieve accurate channel prediction without any frequency-domain conversion. We obtain robust communication performance up to a forecasting time length of 8 ms. These results suggest a high capability of spin-wave reservoir computing in the application of channel prediction and its promising potential in other possible time-sequential computational tasks.

We propose a spin-wave antenna structure that penetrates a garnet film, which we named film-penetrating transducers (FPTs). FPTs possess a zero-dimensional feature that allows flexible placements and sufficient numbers of input/output electrodes for spin-wave reservoir computing. We first explain the structure and operation of FPTs. Then, we numerically construct and analyze a basic spin-wave reservoir chip model. We obtain intricate patterns of spin-wave propagation and interference that reflect the high dimensionality of the system. We also demonstrate the nonlinearity of the output electrical signal from an FPT detector of the system. Our results strongly suggest that FPTs preserve the important properties of a physical reservoir while ensuring large degrees of freedom for both the arrangements and amounts of spin-wave exciters and detectors.

In this paper, we propose reconstructive reservoir computing (RRC) to detect anomaly in time-series signals. In the RRC, an echo state network (ESN) learns to reconstruct normal input signals fed to its input terminals. Since it fails to reconstruct abnormal signals, RRC can detect anomaly based on its reconstruction error. It is shown for the first time that an ESN reconstructs time-series signals effectively for anomaly detection while we already know that it can realize anomaly detection by forecasting. Experiments demonstrate that the RRC works for anomaly detection effectively. Though forecasting errors are used in conventional methods for anomaly detection working for time-series signals, we find experimentally that the reconstruction method has an advantage in its larger margin between normal and abnormal errors. We also find that a smaller leaking rate enhances the ability of anomaly detection. In general, reservoir computing has merits of fast training and, consequently, less energy consumption. We can also make reservoir computing work with the use of physical phenomena. Utilizing reservoir computing is really meaningful for these aspects which layered or other types of recurrent neural networks do not have. Anomaly detection by RRC will be an important application of micro physical-reservoir devices in the near future.

In this paper, we consider various schemes of sequence resampling in reservoir computing models for nonlinear time series prediction. These schemes can enrich the features used for training the readout part with batch learning and lead to better prediction performance. To implement these schemes, first, we introduce a modular approach for constructing multi-reservoir ESN models by assembling encoding and decoding modules. The encoding module is composed of a resampling unit, a group-wise reservoir unit, and a collection unit, for extracting various features from a sequence. The decoding module is a linear regressor which is trainable to produce desired outputs. Then, we propose three novel multi-reservoir ESN models, DeepESN with Every-layer Sequence Resampling (DeepESN-ESR), DeepESN with Last-layer Sequence Resampling (DeepESN-LSR), and GroupedESN with Input-layer Sequence Resampling (GroupedESN-ISR). These three models provide demonstrations of sequence resampling on multi-reservoir ESN models. Numerical results on five challenging nonlinear time-series prediction tasks show that the proposed models outperform some state-of-the-art multi-reservoir ESN models. An evaluation of computational time shows that our proposed three models require less computational cost for learning than many existing multi-reservoir ESN models in practice. Moreover, a comprehensive comparative analysis reveals that our proposed models are able to memorize longer temporal information and generate richer dynamics from the reservoir states than some existing models. The proposed schemes for extracting various features hidden in a sequence of reservoir states can be widely leveraged in other reservoir computing systems for improving their performance in nonlinear time series prediction.

With the penetration of artificial intelligence (AI) technology into industrial applications, not only computational effectiveness but also computational efficiency in machine learning (ML) methods has been increasingly demanded. Reservoir computing (RC) is an ML framework leveraging a dynamicreservoirfor a nonlinear transformation of sequential inputs and areadoutfor mapping the reservoir state to a desired output. Since only the readout is trained with a simple learning algorithm, RC has attracted much attention as a promising approach to enhance compatibility between high computational performance and low learning cost. In addition, recent studies on physical reservoirs implemented with various physical substrates have boosted the potential of RC in the development of effective and efficient AI hardware. Therefore, it is time to further explore the new frontiers in extremely efficient RC.

Recurrent neural networks (RNNs) are helpful tools for modeling dynamical systems by neuronal populations, but efficiently training RNNs has been a challenging topic. In recent years, a recursive least squares (RLS) based method for modifying all the recurrent connections, called the full-Force method, has been gaining attention as a fast and robust online training rule. This method introduces a second network (called the teacher reservoir) during training to provide suitable target dynamics to all the hidden units of the task-performing network (called the student network). Thanks to the RLS-based approach, the full-FORCE method can be applied to training continuous-time networks and spiking neural networks. In this study, we propose a generalized version of the full-FORCE method: the partial-FORCE method. In the proposed method, only part of the student network neurons (called supervised neurons) is supervised by only part of the teacher reservoir neurons (called supervising neurons). As a result of this relaxation, the size of the student network and that of the teacher reservoir can be different, which is biologically plausible as a possible model of the memory transfer in the brain. Furthermore, we numerically show that the partial-FORCE method converges faster and is more robust against variations in parameter values and initial conditions than the full-FORCE method, even without the price of computational cost.

Reservoir computing is a machine learning framework derived from a special type of recurrent neural network. Following recent advances in physical reservoir computing, some reservoir computing devices are thought to be promising as energy-efficient machine learning hardware for real-time information processing. To realize efficient online learning with low-power reservoir computing devices, it is beneficial to develop fast convergence learning methods with simpler operations. This study proposes a training method located in the middle between the recursive least squares (RLS) method and the least mean squares (LMS) method, which are standard online learning methods for reservoir computing models. The RLS method converges fast but requires updates of a huge matrix called a gain matrix, whereas the LMS method does not use a gain matrix but converges very slow. On the other hand, the proposed method called a transfer-RLS method does not require updates of the gain matrix in the main-training phase by updating that in advance (i.e., in a pre-training phase). As a result, the transfer-RLS method can work with simpler operations than the original RLS method without sacrificing much convergence speed. We numerically and analytically show that the transfer-RLS method converges much faster than the LMS method. Furthermore, we show that a modified version of the transfer-RLS method (called transfer-FORCE learning) can be applied to the first-order reduced and controlled error (FORCE) learning for a reservoir computing model with a closed-loop, which is challenging to train.

This paper investigates the spatial distribution of information effective for function learning in a spin-wave reservoir-computing garnet chip. We map the neural weights of a readout neuron virtually connected massively and densely to the reservoir chip. We find that the spatial weight distribution shows wavefront-like lines, suggesting the importance of concurrent and time-different interferences of the spin waves. We also estimate the size of reservoir output electrodes required for the proper information extraction. These results are significantly useful for designing spin reservoir chips in the near future energy efficient devices.

Nonlinear time-series prediction is one of the challenging tasks in machine learning. Recurrent neural networks and their variants have been successful in such a task owing to its ability of storing past inputs in their dynamical states. Echo state networks (ESNs) are a special type of recurrent neural networks, which are capable of high-speed learning. To develop this computational scheme, we propose an HP-ESN method which combines ESNs with a preprocessing based on the Hodrick- Prescott (HP) filter. This filter extracts different components from a single time-series data. The extracted components are processed by ESNs. We show that the proposed method yields better prediction performance compared with other state-of- the-art ESN-based methods in prediction tasks with real-world time-series data. We also demonstrate that the computational performance depends on the setting of the smoothing parameter and the number of decompositions by the HP filter.

Nonlinear time-series prediction is one of the challenging topics in machine learning due to complex non-stationarity in the temporal dynamics. Many recurrent neural network models have been proposed for enhancing the prediction accuracy in time-series prediction tasks. Echo state networks (ESNs) are a variant of recurrent neural networks, which have great potential for addressing machine learning tasks with a very low learning cost. However, the existing ESN-based models have used only single-span features to our best knowledge. In this study, we propose two deep ESN models incorporating multi-span features to improve the prediction performance. We show that the two deep ESN models yield better prediction performance compared to the other state-of-the-art ESN-based methods in benchmark time-series prediction tasks with three models: the Lorenz system, the Mackey-Glass system, and the NARMA-10 system. Our analyses illustrate that deeper structures decrease the multicollinearity of the extracted features and thus contribute to improved performance. The presented results suggest that the proposed models contribute to the development of artificial intelligence for temporal information processing.

The development of hardware neural networks, including neuromorphic hardware, has been accelerated over the past few years. However, it is challenging to operate very large-scale neural networks with low-power hardware devices, partly due to signal transmissions through a massive number of interconnections. Our aim is to deal with the issue of communication cost from an algorithmic viewpoint and study learning algorithms for energy-efficient information processing. Here, we consider two approaches to finding spatially arranged sparse recurrent neural networks with the high cost-performance ratio for associative memory. In the first approach following classical methods, we focus on sparse modular network structures inspired by biological brain networks and examine their storage capacity under an iterative learning rule. We show that incorporating long-range intermodule connections into purely modular networks can enhance the cost-performance ratio. In the second approach, we formulate for the first time an optimization problem where the network sparsity is maximized under the constraints imposed by a pattern embedding condition. We show that there is a tradeoff between the interconnection cost and the computational performance in the optimized networks. We demonstrate that the optimized networks can achieve a better cost-performance ratio compared with those considered in the first approach. We show the effectiveness of the optimization approach mainly using binary patterns and apply it also to gray-scale image restoration. Our results suggest that the presented approaches are useful in seeking more sparse and less costly connectivity of neural networks for the enhancement of energy efficiency in hardware neural networks.

Reservoir Computing (RC) is a framework based on recurrent neural networks for high-speed learning and has attracted much attention. RC has been applied to a variety of temporal recognition tasks. Especially, Jaeger showed that the Echo State Network (ESN), which is one of the RC models, was effective for chaotic time series prediction tasks. However, there are two inevitable problems in nonlinear time series prediction using the standard ESN. One is that its prediction ability reaches a saturation point as the reservoir size is increased. The other is that its prediction ability depends heavily on hyperparameter values. In this paper, we propose a multi-step learning ESN to solve these problems. The proposed system has multiple reservoirs and the prediction error of one ESN-based predictor is corrected by another subsequent predictor. We demonstrate the effectiveness of the proposed method in two nonlinear time series prediction tasks. Another experiment using Lyapunov exponents suggests that the performance of the proposed method is robust against changes in hyperparameter values. In addition, we clarify the characteristic of the proposed method with regard to nonlinearity and memory using simple function approximation tasks.

Reservoir computing is a computational framework which is originally based on software recurrent neural networks and recently achieved with physical systems as well. In our previous paper [Nakane et al., IEEE ACCESS vol. 6, p. 4462, 2018], we have proposed a spin-wave-based reservoir computing device with multiple input/output electrodes, and have demonstrated its high generalization ability in the estimation of input-signal parameters performed by the spin-wave-based reservoir computing. To successfully execute many types of estimation tasks with machine learning, it is necessary to investigate fundamental properties of spin-wave-based reservoir computing, particularly the relation between its input and output. From this background, the purposes of this work are to demonstrate a different estimation task with pulse input signals and to analyze the properties of spin waves which have important roles in the task. We first describe our approach to obtain spin waves with the features useful for reservoir computing, by considering the fundamental properties of spin waves and feasible device technologies. Then, we investigate detailed characteristics of locally-excited spin waves in a garnet film by micromagnetics simulation. Using the resultant spin waves, we demonstrate a pulse interval estimation task, and achieve a high diversity in the time-sequential signals generated by the spin-wave-based reservoir. The spin-wave-based device is a highly promising hardware for next-generation machine-learning electronics.

Reservoir computing is a computational framework suited for temporal/sequential data processing. It is derived from several recurrent neural network models, including echo state networks and liquid state machines. A reservoir computing system consists of a reservoir for mapping inputs into a high-dimensional space and a readout for pattern analysis from the high-dimensional states in the reservoir. The reservoir is fixed and only the readout is trained with a simple method such as linear regression and classification. Thus, the major advantage of reservoir computing compared to other recurrent neural networks is fast learning, resulting in low training cost. Another advantage is that the reservoir without adaptive updating is amenable to hardware implementation using a variety of physical systems, substrates, and devices. In fact, such physical reservoir computing has attracted increasing attention in diverse fields of research. The purpose of this review is to provide an overview of recent advances in physical reservoir computing by classifying them according to the type of the reservoir. We discuss the current issues and perspectives related to physical reservoir computing, in order to further expand its practical applications and develop next-generation machine learning systems.

Reservoir computing has attracted much attention for its easy training process as well as its ability to deal with temporal data. A reservoir computing system consists of a reservoir part represented as a sparsely connected recurrent neural network and a readout part represented as a simple regression model. In machine learning tasks, the reservoir part is fixed and only the readout part is trained. Although reservoir computing has been mainly applied to time series prediction and recognition, it can be applied to image recognition as well by considering an image data as a sequence of pixel values. However, to achieve a high performance in image recognition with raw image data, a large-scale reservoir including a large number of neurons is required. This is a bottleneck in terms of computer memory and computational cost. To overcome this bottleneck, we propose a new method which combines reservoir computing with untrained convolutional neural networks. We use an untrained convolutional neural network to transform raw image data into a set of smaller feature maps in a preprocessing step of the reservoir computing. We demonstrate that our method achieves a high classification accuracy in an image recognition task with a much smaller number of trainable parameters compared with a previous study.

This study explores efficient structures of artificial neural networks for associative memories. Motivated by the real brain structure and the demand of energy efficiency in hardware implementation, we consider neural networks with sparse modular structures. Numerical experiments are performed to clarify how the storage capacity of associative memory depends on regularity and randomness of the network structures. We first show that a fully regularized network, suited for design of hardware, has poor recall performance and a fully random network, undesired for hardware implementation, yields excellent recall performance. For seeking a network structure with good performance and high implementability, we consider four different modular networks constructed based on different combinations of regularity and randomness. From the results of associative memory tests for these networks, we find that the combination of random intramodule connections and regular intermodule connections works better than the other cases. Our results suggest that the parallel usage of regularity and randomness in network structures could be beneficial for developing energy-efficient neural networks.

A widely used complex-valued activation function for complex-valued multistate Hopfield networks is revealed to be essentially based on a multilevel step function. By replacing the multilevel step function with other multilevel characteristics, we present two alternative complex-valued activation functions. One is based on a multilevel sigmoid function, while the other on a characteristic of a multistate bifurcating neuron. Numerical experiments show that both modifications to the complex-valued activation function bring about improvements in network performance for a multistate associative memory. The advantage of the proposed networks over the complex-valued Hopfield networks with the multilevel step function is more outstanding when a complex-valued neuron represents a larger number of multivalued states. Further, the performance of the proposed networks in reconstructing noisy 256 gray-level images is demonstrated in comparison with other recent associative memories to clarify their advantages and disadvantages.

The complex-signum function has been widely used as an activation function in complex-valued recurrent neural networks for multistate associative memory. This paper presents two alternative activation functions with circularity. One is the complex-sigmoid function based on a multilevel sigmoid function defined on a circle. The other is a characteristic of a bifurcating neuron represented by a circle map. The performance of the complex-valued neural networks with the two kinds of activation functions is investigated in multistate associative memory tests. In both networks, the connection weights to store the memory patterns are determined by the generalized projection rule. We also demonstrate gray-level image reconstruction as a possible application of the proposed methods.