To obtain better value estimation in reinforcement learning, we propose a novel algorithm based on the double actor-critic framework with temporal difference error-driven regularization, abbreviated as TDDR. TDDR employs double actors, with each actor paired with a critic, thereby fully leveraging the advantages of double critics. Additionally, TDDR introduces an innovative critic regularization architecture. Compared to classical deterministic policy gradient-based algorithms that lack a double actor-critic structure, TDDR provides superior estimation. Moreover, unlike existing algorithms with double actor-critic frameworks, TDDR does not introduce any additional hyperparameters, significantly simplifying the design and implementation process. The convergence of the proposed TDDR to the optimal value is analyzed under random updating and simultaneous updating patterns. Extensive experiments on various tasks, including MuJoCo and Box2D, demonstrate that TDDR performs competitively against 13 algorithms, including both benchmarks and state-of-the-art methods. It also achieves statistically significant performance gains across several environments.

Actor-critic (AC) algorithms are model-free deep reinforcement learning techniques that have consistently demonstrated effectiveness across various domains. Enhancing exploration (action entropy) and exploitation (expected return) through more efficient sample utilization is pivotal to their success. A key strategy for a learning algorithm is to intelligently navigate the environment's state space, prioritizing the exploration of rarely visited states over frequently encountered ones. However, conventional approaches rarely quantify a novel state's utility for policy learning, which can lead to inefficient exploration. To address this, we propose an innovative approach to bolster exploration by employing an intrinsic reward based on a state's novelty and the potential benefits of exploring that state, which we term plausible novelty. Our method seamlessly integrates with off-policy AC algorithms. By incentivizing the exploration of plausibly novel states, AC algorithms can achieve substantial improvements in sample efficiency and overall training performance. Empirical results demonstrate 19% improvement in training return and 30% reduction in standard deviation, averaged across comparisons of three benchmark algorithm pairs in five different environments.

Model-based reinforcement learning methods enhance sample efficiency by generating synthetic data during training. However, modeling errors can undermine training, leading to failures when applied to the actual environment, especially due to discrepancies in the learned dynamics. In this paper, we propose a new ensemble of double surrogate models created using symbolic regression to uncover the fundamental physical principles governing system behavior, thereby enabling more data-efficient real-world applications. Symbolic regression enhances model interpretability by producing straightforward models that generalize well with limited data. These models interact with reinforcement learning algorithms, with interactions occurring solely within the synthetic models, significantly reducing the need for real experimental data. The structure of the double surrogate models mitigates model bias, preventing agents from exploiting inaccuracies in the environment that could lead to poor performance. Our approach demonstrates comparable training performance when validated in real environments, requiring less than 1 % of the experimental data typically needed for conventional reinforcement learning algorithms.

This paper investigates the prescribed-time cooperative output regulation (PTCOR) for a class of linear heterogeneous multi-agent systems (MASs) under directed communication graphs. As a special case of PTCOR, the necessary and sufficient condition for prescribed-time output regulation of an individual system is first explored, whereas only sufficient conditions are developed in the literature. A PTCOR algorithm is subsequently developed, composed of prescribed-time distributed observers, local state observers, and tracking controllers, utilizing a distributed feedforward method. This approach converts the PTCOR problem into the prescribed-time stabilization problem of a cascaded subsystem. The criterion for the prescribed-time stabilization of the cascaded system is proposed, differing from that of traditional asymptotic or finite-time stabilization of a cascaded system. It is proven that the regulated outputs converge to zero within a prescribed time and remain at zero afterward, while all internal signals in the closed-loop MASs are uniformly bounded. Finally, the theoretical results are validated through two numerical examples.

This article addresses the distributed prescribed-time convex optimization (DPTCO) problem for high-order nonlinear multiagent systems (MASs) under undirected connected graphs. A cascade design framework is proposed that divides the DPTCO implementation into distributed optimal trajectory generator design and local reference trajectory tracking controller design. The DPTCO problem is then transformed into the prescribed-time stabilization problem of a cascaded system. Using changing Lyapunov functions and time-varying state transformations with sufficient conditions, we establish criteria for prescribed-time stabilization and prove the boundedness of internal signals in closed-loop MASs. The framework addresses robust DPTCO for chain-integrator MASs with disturbances through the introduction of novel sliding-mode variables and time-varying gains. It also solves adaptive DPTCO for strict-feedback MASs with parameter uncertainty via backstepping method and descending power state transformation. Two numerical examples verify the theoretical results.

Temporal difference (TD) learning is a fundamental technique in reinforcement learning that updates value function estimates for states or state-action pairs using a TD target. This target represents an improved estimate of the true value by incorporating both immediate rewards and the estimated value of subsequent states. We propose an enhanced multistate TD (MSTD) target that utilizes multiple subsequent states for a more accurate value function estimation compared to traditional TD learning, which relies on a single subsequent state. Building on this new MSTD concept, we develop actor-critic algorithms that include the management of replay buffers in two modes and integrate with deep deterministic policy optimization (DDPG) and soft actor-critic (SAC). Numerical experiment results demonstrate that algorithms employing the MSTD target improve learning performance compared to traditional methods. In addition, we analyze the convergence of Q-learning with MSTD.

This article investigates the global robust output regulation of a class of multi-input multi-output (MIMO) time-varying (TV) nonlinear systems with a TV exosystem. Existing linear or nonlinear internal model designs are inadequate for the output regulation problem in TV situations. Therefore, we first transform this control problem into a global robust stabilization problem (GRSP) for a more complex MIMO TV augmented nonlinear system by constructing a TV internal model. Under a series of standard assumptions, this system can be globally stabilized by a recursive state feedback controller. Finally, the proposed algorithm is applied to the disturbance rejection problem of a permanent magnet synchronous motor (PMSM) position servo system, and the experimental results demonstrate its effectiveness.

This article studies fixed-time robust networked observers to estimate the information of a leader system containing unknown parameters. The construction of observers combines the internal model principle and the fixed-time control technique in a network with a directed topology. The observers are further applied to the fixed-time attitude synchronization problem for multiple spacecraft systems whose attitudes are represented by unit quaternion. Both rigorous analysis and numerical simulation demonstrate that the fixed-time synchronization of attitude and angular velocity between multiple spacecrafts and a specified leader system is achieved by the observer-based control design.

In this article, the global event-triggered (ET) funnel tracking control problem is studied for a class of switched nonlinear systems with structural uncertainties, where the solvability of the control problem for each subsystem is not needed. A switching multiple Lyapunov functions (MLFs) method is established, where MLFs are designed to handle switched inverse dynamics, and a switching barrier Lyapunov function is constructed to address switched sampled errors that may compromise system stability. This is achieved alongside a new switching dynamic event-triggering mechanism (DETM). By combining this method with backstepping, a dwell-time state-dependent switching law and an ET funnel controller of each subsystem are constructed, effectively eliminating the issue of the "explosion of complexity" encountered in traditional backstepping without using dynamic surface control or command filters. Additionally, the designed switching DETM ensures that the tracking error always evolves within a performance funnel in any consecutive triggering interval, excluding Zeno behavior, and guaranteeing positive constant lower bounds for two consecutive triggering intervals and any switching interval, respectively. Finally, an example is provided to show the validity of the theoretical results.

Soft actor-critic (SAC) is an off-policy actor-critic (AC) reinforcement learning (RL) algorithm, essentially based on entropy regularization. SAC trains a policy by maximizing the trade-off between expected return and entropy (randomness in the policy). It has achieved the state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. SAC works in an off-policy fashion where data are sampled uniformly from past experiences (stored in a buffer) using which the parameters of the policy and value function networks are updated. We propose certain crucial modifications for boosting the performance of SAC and making it more sample efficient. In our proposed improved SAC (ISAC), we first introduce a new prioritization scheme for selecting better samples from the experience replay (ER) buffer. Second we use a mixture of the prioritized off-policy data with the latest on-policy data for training the policy and value function networks. We compare our approach with the vanilla SAC and some recent variants of SAC and show that our approach outperforms the said algorithmic benchmarks. It is comparatively more stable and sample efficient when tested on a number of continuous control tasks in MuJoCo environments.

This article addresses the resilient practical cooperative output regulation problem (RPCORP) for multiagent systems subjected to both denial-of-service (DoS) attacks and actuator faults. Fundamentally different from the existing solutions to RPCORPs, the system parameters considered in this article are unknown to each agent, and a novel data-driven control approach is introduced to handle such an issue. The solution starts with developing resilient distributed observers for each follower in the presence of DoS attacks. Then, a resilient communication mechanism and a time-varying sampling period are introduced to, respectively, ensure the neighbor state is available as soon as attacks disappear and to avoid targeted attacks launched by intelligent attackers. Furthermore, a model-based fault-tolerant and resilient controller is designed based on the Lyapunov approach and the output regulation theory. In order to remove the reliance on system parameters, we leverage a new data-driven algorithm to learn controller parameters via the collected data. Rigorous analysis shows that the closed-loop system can resiliently achieve practical cooperative output regulation. Finally, a simulation example is given to illustrate the effectiveness of the achieved results.

Trajectory optimization is a fundamental stochastic optimal control (SOC) problem. This article deals with a trajectory optimization approach for dynamical systems subject to measurement noise that can be fitted into linear time-varying stochastic models. Exact/complete solutions to these kind of control problems have been deemed analytically intractable in literature because they come under the category of partially observable Markov decision processes (MDPs). Therefore, effective solutions with reasonable approximations are widely sought for. We propose a reformulation of stochastic control in a reinforcement learning setting. This type of formulation assimilates the benefits of conventional optimal control procedure, with the advantages of maximum likelihood approaches. Finally, an iterative trajectory optimization paradigm called as SOC-expectation maximization (SOC-EM) is put forth. This trajectory optimization procedure exhibits better performance in terms of reduction in cumulative cost-to-go which is proven both theoretically and empirically. Furthermore, we also provide novel theoretical work which is related to uniqueness of control parameter estimates. Analysis of the control covariance matrix is presented, which handles stochasticity through efficiently balancing exploration and exploitation.

Model-based design is an important method of addressing problems associated with designing complex control systems. For complex dynamic systems in the presence of uncertainties, the modeling process from the first principles becomes extremely tedious and simplification in mechanism and parameter measurement may result in model inaccuracy. On the contrary, machine learning has the characteristic of fitting complicated equations, which makes it widely used in the research of model identification. However, it only brings a black-box model where the design schemes based on an analytical model cannot be applied. In this article, a simple and novel scheme for modeling and control of robotic manipulators is proposed; without prior knowledge, a dynamic model in an analytical form is obtained from artificially excited training data using the symbolic regression technique, and then, a controller is designed based on the dynamic model. Due to the ingenious experimental design, on one hand, the amount of training data is far less than the system identification method by machine learning. On the other hand, a decoupling feature is used in the model that greatly simplifies controller design. The experimental results on two-degree of freedom (DOF) and 6-DOF robotic manipulator simulators verify that the scheme is feasible and effective.

Recent advances in Multi-agent Reinforcement Learning (MARL) have made it possible to implement various tasks in cooperative as well as competitive scenarios through trial and error, and deep neural networks. These successes motivate us to bring the mechanism of MARL into the Multi-agent Resilient Consensus (MARC) problem that studies the consensus problem in a network of agents with faulty ones. Relying on the natural characteristics of the system goal, the key component in MARL, reward function, can thus be directly constructed via the relative distance among agents. Firstly, we apply Deep Deterministic Policy Gradient (DDPG) on each single agent to train and learn adjacent weights of neighboring agents in a distributed manner, that we call Distributed-DDPG (D-DDPG), so as to minimize the weights from suspicious agents and eliminate the corresponding influences. Secondly, to get rid of neural networks and their time-consuming training process, a Q-learning based algorithm, called Q-consensus, is further presented by building a proper reward function and a credibility function for each pair of neighboring agents so that the adjacent weights can update in an adaptive way. The experimental results indicate that both algorithms perform well with appearance of constant and/or random faulty agents, yet the Q-consensus algorithm outperforms the faulty ones running D-DDPG. Compared to the traditional resilient consensus strategies, e.g., Weighted-Mean-Subsequence-Reduced (W-MSR) or trustworthiness analysis, the proposed Q-consensus algorithm has greatly relaxed the topology requirements, as well as reduced the storage and computation loads. Finally, a smart-car hardware platform consisting of six vehicles is used to verify the effectiveness of the Q-consensus algorithm by achieving resilient velocity synchronization.

In this paper, a multi-hypothesis square-root cubature Kalman particle filter (MH-SRCKPF) is proposed for speaker tracking in noisy and reverberant environments with distributed microphone arrays. The conventional cubature Kalman particle filter (CKPF) uses the cubature Kalman filter (CKF) to generate its proposal for particle sampling. Such a proposal incorporates only one observation from a certain localization function for the state estimation, which is vulnerable to noise or reverberation, yielding the degraded tracking performance. To tackle the problem, by incorporating multiple possible observations into CKF for the proposal, a multi-hypothesis CKPF (MH-CKPF) algorithm is first developed. Furthermore, to improve the numerical stability, an MH-SRCKPF algorithm is developed, where the state estimate and the square root of the error covariance are propagated at each time. Finally, the MH-SRCKPF is applied to the speaker tracking problems in distributed microphone arrays. Experimental results demonstrate that the proposed MH-SRCKPF outperforms the competing methods in the presence of noise and reverberation. Meanwhile, by propagating the square root of the state covariance, the proposed method exhibits attractive numerical characteristics.

This paper focuses on constructing genotypic predictors for antiretroviral drug susceptibility of HIV. To this end, a method to recover the largest elements of an unknown vector in a least squares problem is developed. The proposed method introduces two novel ideas. The first idea is a novel forward stepwise selection procedure based on the magnitude of the estimates of the candidate variables. To implement this newly introduced procedure, we revise Tikhonov regularisation from a sparse representations' perspective. This analysis leads us to the second novel idea in the paper, which is the development of a new method to recover the largest elements of the unknown vector in the least squares problem. The method implements a sequence of Tikhonov regularisation problems which aim to recover the largest of the remaining elements of the unknown vector. Additionally, we derive sufficient conditions that ensure the recovery of the largest elements of the unknown vector. We perform numerical studies using simulated data and data from the Stanford HIV resistance database. The performance of the proposed method is compared against a state-of-the-art method.

Convolutional Neural Networks (CNNs) are currently the most prominent deep neural network models and have been used with great success for image classification and other applications. The performance of CNNs depends on their architecture and hyperparameter settings. Early CNN models like LeNet and AlexNet were manually designed by experienced researchers. The empirical design and optimisation of a new CNN architecture require a lot of expertise and can be very time-consuming. In this paper, we propose a genetic algorithm that can, for a given image processing task, efficiently explore a defined space of potentially suitable CNN architectures and simultaneously optimise their hyperparameters. We named this fast automatic optimisation model fast-CNN and employed it to find competitive CNN architectures for image classification on CIFAR10. In a series of comparative simulation experiments we could demonstrate that the network designed by fast-CNN achieved nearly as good accuracy as some of the other best network models available but fast-CNN took significantly less time to evolve. The trained fast-CNN network model also generalised well to CIFAR100.

This paper explores a cooperative practical output regulation problem for a class of heterogeneous multiagent nonlinear systems by event-based output feedback. Specifically, we shall restrict our attention to the situation of sampled-data-based local measurements. As usual, due to agents heterogeneity, we first convert the problem into a stabilization one for the so-called augmented system, composed of the agent systems and suitably designed continuous-time internal models. Then, we show that this stabilization can be solved by measurement feedback. It finally allows us to establish a valid event-based protocol, leading to a global practical stability property and meanwhile guaranteeing Zeno-free condition.

This letter studies a single-user power splitting-based multiple-input multiple-output system for simultaneous wireless information and power transfer. We aim to maximize the harvested power by joint design of transmit signal covariance matrix and receive power splitting factor under both a system rate constraint and a total power constraint. The harvested power maximization problem is difficult to solve due mainly to the nonconcave objective and the nonlinear coupling of design variables in the constraints. To tackle these challenges, we first derive a good approximation of the problem by ignoring some negligible noise terms and then further simplify it to a more tractable form by well exploiting the problem structure. Based on the Frank-Wolfe algorithm, we propose a simple yet efficient iterative algorithm to address the resulting problem. Numerical results validate the efficiency of the proposed algorithm.

This paper investigates a group of multiagents moving on a 2-D plane with a constant speed but maneuverable headings. It is called a minimal control multiagent model (MCMA) when each agent employs a static decentralized control law that relies on its neighbors' relative positions with respect to its local reference frame. In other words, the control law does not involve any complicated velocity measurement or estimation mechanism. Various minimal multiagent models have been investigated and extensively simulated in terms of their collaborative behaviors. This paper, for the first time, gives rigorous theoretical proofs for the functionalities of an MCMA model in both collision avoidance and velocity alignment.

The conventional robust output regulation problem aims to achieve reference tracking and disturbance rejection in the presence of system uncertainties while the references and disturbances are generated by an autonomous exosystem. When the exosystem is perturbed by an external event, a novel robust perturbed output regulation problem is formulated and solved in this paper. The formulation arises from a class of synchronization problem of multiple agents. With the aid of a new reference model-based control framework, the proposed solution to the robust perturbed output regulation problem leads to a decentralized control algorithm for synchronization of multiple agents of nonlinear heterogeneous dynamics.

This paper reviews some main results and progress in distributed multi-agent coordination from a graph Laplacian perspective. Distributed multi-agent coordination has been a very active subject studied extensively by the systems and control community in last decades, including distributed consensus, formation control, sensor localization, distributed optimization, etc. The aim of this paper is to provide both a comprehensive survey of existing literature in distributed multi-agent coordination and a new perspective in terms of graph Laplacian to categorize the fundamental mechanisms for distributed coordination. For different types of graph Laplacians, we summarize their inherent coordination features and specific research issues. This paper also highlights several promising research directions along with some open problems that are deemed important for future study.

For a multiagent system in free space, the agents are required to generate sufficiently large cohesive force for network connectivity preservation and sufficiently large repulsive force for collision avoidance. This paper gives an energy function based approach for estimating the control force in a general setting. In particular, the force estimated for network connectivity preservation and collision avoidance is separated from the force for other collective behavior of the agents. Moreover, the estimation approach is applied in three typical collective control scenarios including swarming, flocking, and flocking without velocity measurement.

A fastest consensus problem of topology fixed networks has been formulated as an optimal linear iteration problem and efficiently solved in the literature. Considering a kind of predictive mechanism, we show that the consensus evolution can be further accelerated while physically maintaining the network topology. The underlying mechanism is that an effective prediction is able to induce a network with a virtually denser topology. With this topology, an even faster consensus is expected to occur. The result is motivated by the predictive mechanism widely existing in natural systems.

Typical animal locomotion is achieved by the rhythmical undulation of its body segments while interacting with its environment. It inspires the mechanical design of multilink locomotors. With different postures, a multilink system may present different locomotion gaits. Recently, a so-called natural oscillation gait was studied for multilink systems, and a class of biologically inspired controllers was designed for the achievement of the gait. In this paper, the theoretical design is experimentally applied on a mechanical multilink testbed of two posture configurations in rayfish-like flapping-wing motion and snake-like serpentine motion. The effectiveness of the design is cross examined by theoretical analysis, numerical simulation, and experiments.

The connectivity of a group of agents in a flocking scenario is caused either by individual's local cohesion interaction mechanism or by external boundary constraints. The latter case is particularly interesting when an individual's cohesion ability is not reliable due to the limitation of communication range. The effect of external boundary constraints on the connectivity property of multiagents has been intensively investigated in natural observation and engineering simulation. A theoretical analysis is given in this paper which reveals that a group of agents in a bounded plane can be almost always jointly connected and hence form a complete flock.

Robust continuous-time control for nonlinear output feedback systems has been well studied in literature. In particular, the robust controller is based on the high gain domination approach. It is interesting to further study the digital implementation of the controller in a practical situation when sampled data with possible delay are available for feedback design. A fundamental difficulty lies in the fact that high robustness requires a high control gain which is however constrained by sampling bandwidth. In this paper, we apply a recently established sampled-data control framework to reveal the quantitative tradeoff between robustness and sampling bandwidth in nonlinear systems.

Precision control of piezoelectric motor nanopositioning stages is widely used in a variety of nano-manufacturing equipments. But due to the hysteresis nonlinearity with input saturation, it is challenging to design an ultrafast output feedback controller with large region of closed-loop stability. To address this problem, we developed a dual-mode nonlinear model predictive control (NMPC) method, in which an optimal input profile found by solving an open-loop optimal control problem drives the nonlinear system state into the terminal invariant set; afterwards a linear output-feedback controller steers the state to the origin asymptotically. In contrast to the classical output-feedback controller, the settling time is effectively decreased and the closed-loop stable region is substantially increased by the present NMPC with almost no loss of the nanopositioning accuracy. Finally, the feasibility and superiority of the proposed switching control method are examined by extensive experiments on a Physik Instrumente P-563.3CL triple-axis nanopositioning stage.

A novel individual-based alignmen/repulsion algorithm is proposed in this paper for a flock of multiple agents. With this algorithm, each individual repels its sufficiently close neighbors and aligns to the average velocity of its neighbors with moderate distances. In both mathematical analysis and numerical simulation, we prove that the algorithm guarantees an uncrowded flocking behavior with asymptotical velocity synchronization when sufficiently intensive communication exists within the agents. Moreover, we provide the conditions for collision avoidance along the whole transient procedure. The proposed flocking model has its references in natural collective behaviors like escaping panic and traffic jam motions.

The small gain theorem as established in [Z.P. Jiang and I. Mareels, 1997] and [Z.P. Jiang et al., 1994] has been given in terms of a quite general inter-connection of two nonlinear subsystems. This paper gives a more clear-cut version and a more straightforward proof for the small gain theorem for a simpler inter-connection of two nonlinear subsystems by using a different characterization of the ISS concept. Moreover, we have also incorporated some type of model uncertainties into the system description, thus making this special version directly applicable to the global robust stabilization and output regulation problems.