Reservoir computing networks (RCNs) have been recognized as a popular machine learning tool for modeling the temporal evolution of dynamic data due to their close relationship with dynamical systems. A defining characteristic of RCNs is their fixed (training parameter-free) hidden layers, which offers significant computational benefits. However, this feature also introduces adverse impacts, such as increased warm-up time, limited long-term memory, and sensitivity to hyperparameters. To balance these advantages and disadvantages and expand the application domains of RCNs, we develop a novel deep reservoir computing network (DRCN) architecture that integrates control-theoretic concepts and techniques into RCNs. This architecture is designed as a cascade of shallow RCNs and is represented as a piecewise time-invariant control system. We further propose a layer-by-layer training strategy for the DRCN, resulting in an iterative deep learning algorithm for modeling dynamical systems. This enables us to exploit the DRCN as a generative model to generate output-of-sample data using the learned dynamical systems. The performance and efficiency of the DRCN-based dynamic generative model are demonstrated through various learning problems arising from time-series analysis and control systems, using both synthetic and real-world datasets.

Interest in reinforcement learning (RL) for large-scale systems, comprising extensive populations of intelligent agents interacting with heterogeneous environments, has surged significantly across diverse scientific domains in recent years. However, the large-scale nature of these systems often leads to high computational costs or reduced performance for most state-of-the-art RL techniques. To address these challenges, we propose a novel RL architecture and derive effective algorithms to learn optimal policies for arbitrarily large systems of agents. In our formulation, we model such systems as parameterized control systems defined on an infinite-dimensional function space. We then develop a moment kernel transform that maps the parameterized system and the value function into a reproducing kernel Hilbert space. This transformation generates a sequence of finite-dimensional moment representations for the RL problem, organized into a filtrated structure. Leveraging this RL filtration, we develop a hierarchical algorithm for learning optimal policies for the infinite-dimensional parameterized system. To enhance the algorithm's efficiency, we incorporate early stopping at each hierarchy, demonstrating the fast convergence property of the algorithm through the construction of a convergent spectral sequence. The performance and efficiency of the proposed algorithm are validated using practical examples in engineering and quantum systems.

Time series data with missing entries are ubiquitous in a broad spectrum of practical and clinical applications, from climatology and cell biology to personalized medicine. This undesired structure arising either due to undesired artifacts (e.g., noise) or by design (e.g., asynchronous or aperiodic sampling in distributed sensors) results in irregularity in the temporal dimension and forms a bottleneck in data mining. Although extensive data science approaches have been proposed to address learning problems involving irregular data, the emphasis was largely placed on filling in the missing entries via interpolation and binning, or the methods were tailored to specific data analytic tasks. In this article, we develop a reservoir computing (RC)-based iterative learning method for recovering missing data in irregular time series generated by dynamical systems and networks. In particular, we formulate this learning task as a fixed-point iterative learning problem and develop a training procedure using an RC network (RCN). We find that when the irregular time series has "sufficient" samples to train an RCN within a tolerant training error then the missing samples in the time series can be recovered systematically. We also derive sufficient conditions with respect to the choices of the reservoir parameters that guarantee the convergence of the iterative procedure. We present several numerical experiments to demonstrate the efficacy of the developed iterative RCN approach. Specifically, we illustrate the capability of our approach to recover missing data in irregular time series generated by chaotic Rössler and Kuramoto-Sivashinsky (KS) systems. Finally, we also report the results of incorporating our approach in an irregular medical data classification task.

Problems involving controlling the collective behavior of a population of structurally similar dynamical systems, the so-called ensemble control, arise in diverse emerging applications and pose a grand challenge in systems science and control engineering. Owing to the severely under-actuated nature and the difficulty of placing large-scale sensor networks, ensemble systems are limited to being actuated and monitored at the population level. Moreover, mathematical models describing the dynamics of ensemble systems are often elusive. Therefore, it is essential to design broadcast controls that excite the entire population in such a way that the heterogeneity in system dynamics is robustly compensated. In this article, we propose a reinforcement learning (RL)-based data-driven control framework incorporating population-level aggregated measurement data to learn a global control signal for steering a dynamic population in the desired manner. In particular, we introduce the notion of ensemble moments induced by aggregated measurements and derive the associated moment system to the original ensemble system. Then, using the moment system, we learn an approximation of optimal value functions and the associated policies in terms of ensemble moments through RL. We illustrate the feasibility and scalability of the proposed moment-based approach via numerical experiments using a population of linear, bilinear, and nonlinear dynamic ensemble systems. We report that the proposed method achieves the desired control objectives of various ensemble control tasks and obtains significantly better averaged-reward when compared with three existing methods.

The reservoir computing networks (RCNs) have been successfully employed as a tool in learning and complex decision-making tasks. Despite their efficiency and low training cost, practical applications of RCNs rely heavily on empirical design. In this article, we develop an algorithm to design RCNs using the realization theory of linear dynamical systems. In particular, we introduce the notion of α -stable realization and provide an efficient approach to prune the size of a linear RCN without deteriorating the training accuracy. Furthermore, we derive a necessary and sufficient condition on the irreducibility of the number of hidden nodes in linear RCNs based on the concepts of controllability and observability from systems theory. Leveraging the linear RCN design, we provide a tractable procedure to realize RCNs with nonlinear activation functions. We present numerical experiments on forecasting time-delay systems and chaotic systems to validate the proposed RCN design methods and demonstrate their efficacy.

An imposing task for a reinforcement learning agent in an uncertain environment is to expeditiously learn a policy or a sequence of actions, with which it can achieve the desired goal. In this article, we present an incremental model learning scheme to reconstruct the model of a stochastic environment. In the proposed learning scheme, we introduce a clustering algorithm to assimilate the model information and estimate the probability for each state transition. In addition, utilizing the reconstructed model, we present an experience replay strategy to create virtual interactive experiences by incorporating a balance between exploration and exploitation, which greatly accelerates learning and enables planning. Furthermore, we extend the proposed learning scheme for a multiagent framework to decrease the effort required for exploration and to reduce the learning time in a large environment. In this multiagent framework, we introduce a knowledge-sharing algorithm to share the reconstructed model information among the different agents, as needed, and develop a computationally efficient knowledge fusing mechanism to fuse the knowledge acquired using the agents' own experience with the knowledge received from its teammates. Finally, the simulation results with comparative analysis are provided to demonstrate the efficacy of the proposed methods in the complex learning tasks.

In this letter, we investigate the fundamental limits on how the interspike time of a neuron oscillator can be perturbed by the application of a bounded external control input (a current stimulus) with zero net electric charge accumulation. We use phase models to study the dynamics of neurons and derive charge-balanced controls that achieve the minimum and maximum interspike times for a given bound on the control amplitude. Our derivation is valid for any arbitrary shape of the phase response curve and for any value of the given control amplitude bound. In addition, we characterize the change in the structures of the charge-balanced time-optimal controls with the allowable control amplitude. We demonstrate the applicability of the derived optimal control laws by applying them to mathematically ideal and experimentally observed neuron phase models, including the widely studied Hodgkin-Huxley phase model, and by verifying them with the corresponding original full state-space models. This work addresses a fundamental problem in the field of neural control and provides a theoretical investigation to the optimal control of oscillatory systems.