This article investigates the optimal control problem of a stochastic nonlinear strict-feedback system subjected to both Wiener and Poisson noises. Since the strict-feedback system does not satisfy the matching condition and involves unknown nonlinear terms and unmeasurable stochastic noises in modeling, an optimized backstepping (OB) technique based on the reinforcement learning (RL) strategy is adopted to design the controller within the identifier–critic–actor architecture. However, the OB technique needs to construct all the virtual and actual controllers as the optimal solutions of their respective subsystems, which inevitably increases the complexity of the algorithm. To alleviate this situation, a novel RL method is proposed, so that the optimized control is unaffected by the disturbances induced by both Wiener and Poisson noises. In addition, an adaptive neural network identifier is incorporated into the RL framework to ensure that the proposed control scheme can be smoothly applied to the unknown nonlinear dynamic system. Finally, a vehicle tracking control example is presented to demonstrate the effectiveness of the proposed method.

It is difficult to ensure tracking performance even for general nonlinear systems under the condition that only intermittent output signals are used. This paper studies the problem of adaptive fuzzy consensus tracking for a class of mismatched nonlinear strict-feedback multi-agent systems (MASs) by using intermittent output signals. Firstly, a novel fuzzy state observer using intermittent output states and fuzzy-logic systems is constructed to estimate the unknown state variables of the system. Secondly, dynamic filtering technology is applied to address the issue of virtual controller non-differentiability caused by intermittent output feedback during the backstepping design process. Furthermore, for the situation where the controller is non-differentiable in stability analysis, an alternative solution can be proposed: initially, a distributed continuous control strategy is developed using conventional continuous output signals, and then the output signals in the previous scheme are replaced with event-triggered signals to design a distributed event-triggered control scheme. It is demonstrated that the designed event-triggered control scheme guarantees that all closed-loop system signals remain semi-globally uniformly ultimately bounded (SGUUB), and the distributed consensus tracking errors can converge to a neighborhood of zero. Finally, the effectiveness of the proposed scheme is verified through simulations involving a group of single-link robots.

This article addresses a distributed time-varying optimal formation protocol for a class of second-order uncertain nonlinear dynamic multiagent systems (MASs) based on an adaptive neural network (NN) state observer through the backstepping method and simplified reinforcement learning (RL). Each follower agent is subjected to only local information and measurable partial states due to actual sensor limitations. In view of the distributed optimized formation strategic needs, the uncertain nonlinear dynamics and undetectable states may jointly affect the stability of the time-varying cooperative formation control. Furthermore, focusing on Hamilton-Jacobi-Bellman optimization, it is almost incapable of directly dealing with unknown equations. Above uncertainty and immeasurability processed by adaptive state observer and NN simplified RL are further designed to achieve desired second-order formation configuration at the least cost. The optimization protocol can not only solve the undetectable states and realize the prescribed time-varying formation performance on the premise that all the errors are SGUUB, but also prove the stability and update the critics and actors easily. Through the above-mentioned approaches offer an optimal control scheme to address time-varying formation control. Finally, the validity of the theoretical method is proven by the Lyapunov stability theory and digital simulation.

In this article, the game-based backstepping control method is proposed for the high-order nonlinear multi-agent system with unknown dynamic and input saturation. Reinforcement learning (RL) is employed to get the saddle point solution of the tracking game between each agent and the reference signal for achieving robust control. Specifically, the approximate optimal solution of the established Hamilton-Jacobi-Isaacs (HJI) equation is obtained by policy iteration for each subsystem, and the single network adaptive critic (SNAC) architecture is used to reduce the computational burden. In addition, based on the separation operation of the error term from the derivative of the value function, we achieve the different proportions of the two agents in the game to realize the regulation of the final equilibrium point. Different from the general use of the neural network for system identification, the unknown nonlinear dynamic term is approximated based on the state difference obtained by the command filter. Furthermore, a sufficient condition is established to guarantee that the whole system and each subsystem included are uniformly ultimately bounded. Finally, simulation results are given to show the effectiveness of the proposed method.

In this article, for the single-input–single-output (SISO) nonlinear strict-feedback system, optimized backstepping (OB) control combined with the dynamic surface (DS) technique is developed. OB is to make every subsystem control of backstepping as the optimized one so as to ensure the entire backstepping control being optimized. However, the original design of OB still needs to repeatedly calculate the derivative of virtual controls, as a result, it will inevitably cause the problem of “differential explosion.” In order to alleviate the phenomenon, the OB control is combined with the DS technique. Furthermore, OB control needs to conduct with reinforcement learning (RL) in every backstepping step, hence simplifying the algorithm of RL is very necessary and substantive for achieving the combination. In this work, because the optimized control derives both critic and actor training laws by utilizing a simple positive function instead of the square of approximation of Hamilton–Jacobi–Bellman (HJB) equation, it can obviously simplify the RL algorithm to compare with the traditional optimizing methods. Finally, the feasibility is illustrated via both theory and simulation.

In this article, an optimized leader-following consensus control scheme is proposed for the nonlinear strict-feedback-dynamic multi-agent system by learning from the controlling idea of optimized backstepping technique, which designs the virtual and actual controls of backstepping to be the optimized solution of corresponding subsystems so that the entire backstepping control is optimized. Since this control needs to not only ensure the optimizing system performance but also synchronize the multiple system state variables, it is an interesting and challenging topic. In order to achieve this optimized control, the neural network approximation-based reinforcement learning (RL) is performed under critic-actor architecture. In most of the existing RL-based optimal controls, since both the critic and actor RL updating laws are derived from the negative gradient of square of the Hamilton-Jacobi-Bellman (HJB) equation's approximation, which contains multiple nonlinear terms, their algorithm are inevitably intricate. However, the proposed optimized control derives the RL updating laws from the negative gradient of a simple positive function, which is correlated with the HJB equation; hence, it can be significantly simple in the algorithm. Meanwhile, it can also release two general conditions, known dynamic and persistence excitation, which are required in most of the RL-based optimal controls. Therefore, the proposed optimized scheme can be a natural selection for the high-order nonlinear multi-agent control. Finally, the effectiveness is demonstrated by both theory and simulation.

In this article, an optimized backstepping (OB) control scheme is proposed for a class of stochastic nonlinear strict-feedback systems with unknown dynamics by using reinforcement learning (RL) strategy of identifier-critic-actor architecture, where the identifier aims to compensate the unknown dynamic, the critic aims to evaluate the control performance and to give the feedback to the actor, and the actor aims to perform the control action. The basic control idea is that all virtual controls and the actual control of backstepping are designed as the optimized solution of corresponding subsystems so that the entire backstepping control is optimized. Different from the deterministic system, stochastic system control needs to consider not only the stochastic disturbance depicted by the Wiener process but also the Hessian term in stability analysis. If the backstepping control is developed on the basis of the published RL optimization methods, it will be difficult to be achieved because, on the one hand, RL of these methods are very complex in the algorithm thanks to their critic and actor updating laws deriving from the negative gradient of the square of approximation of Hamilton-Jacobi-Bellman (HJB) equation; on the other hand, these methods require persistence excitation and known dynamic, where persistence excitation is for training adaptive parameters sufficiently. In this research, both critic and actor updating laws are derived from the negative gradient of a simple positive function, which is yielded on the basis of a partial derivative of the HJB equation. As a result, the RL algorithm can be significantly simplified, meanwhile, two requirements of persistence excitation and known dynamic can be released. Therefore, it can be a natural selection for stochastic optimization control. Finally, from two aspects of theory and simulation, it is demonstrated that the proposed control can arrive at the desired system performance.

In this article, a fuzzy logic system (FLS)-based adaptive optimized backstepping control is developed by employing reinforcement learning (RL) strategy for a class of nonlinear strict feedback systems with unmeasured states. For making the virtual and actual controls are optimized solution of the corresponding subsystem, RL of observer-critic-actor architecture based on FLS approximations is constructed in every backstepping step, where the observer aims to estimate the unmeasurable states, and the critic and actor aim to evaluate control performance and perform control behavior, respectively. In the proposed optimized control, on the one hand, the state observer method can avoid to require the design constants making its characteristic polynomial Hurwitz, which is universally demanded in the existing observer methods; on the other hand, the RL is significantly simple in algorithm, because the critic and actor training laws are derived from the negative gradient of a simple positive function, which is produced from the partial derivative of Hamilton–Jacobi–Bellman (HJB) equation, instead of the square of approximated HJB equation. Therefore this optimized scheme can be more easily applied and widely extended. Finally, from two aspects of theory and simulation, it is demonstrated that the desired objective can be fulfilled.

In this article, an adaptive optimized control scheme based on neural networks (NNs) is developed for a class of perturbed strict-feedback nonlinear systems. An optimized backstepping (OB) technique is employed for breaking through the limitation of the matching condition. The disturbance of existing nonlinear systems may degrade system performance or even lead to instability. In order to improve the system's robustness, a disturbance observer is constructed to compensate for the impact coming from the external disturbance. Because the proposed optimized scheme needs to train the adaptive parameters not only for reinforcement learning (RL) but also for the disturbance observer, it will become more challenging no matter designing the control algorithm or deriving the adaptive updating laws. Finally, by virtue of the Lyapunov stability theory, it is proved that all internal signals of the closed-loop systems are semiglobal uniformly ultimately bounded (SGUUB). Simulation results are provided to illustrate the validity of the devised method.

In this article, an optimized tracking control scheme is studied for the quadrotor unmanned aerial vehicle (QUAV) system by combining both reinforcement learning (RL) and the backstepping technique. The RL aims to overcome the difficulty coming from solving the Hamilton–Jacobi–Bellman (HJB) equation, and it is performed via iterating both critic and actor each other, where the critic is for improving the control performance and the actor is for executing the control behavior. In mathematics, a QUAV system is composed of two connected subsystems that are, respectively, modeled by the translational and rotational dynamic equations, which are coupled via a rotation matrix; hence, the optimized tracking scheme is composed of two interconnected individual controls corresponding to the position and attitude, respectively. To achieve the two optimized position and attitude controls, the RL is constructed on the basis of the neural network (NN) approximation of the HJB equation’s solution by utilizing NN’s outstanding function approximation ability. Particularly, the position control is accomplished by introducing an intermediate control because the translational dynamic is an underactuated system. Since the proposed RL algorithm is significantly simple in comparison with the published methods, the optimized QUAV control can be easily executed in practical applications. Finally, the results are demonstrated by a Lyapunov stability analysis and a numerical simulation.

In this article, an optimized leader-follower consensus control is proposed for a class of second-order unknown nonlinear dynamical multiagent system. Different with the first-order multiagent consensus, the second-order case needs to achieve the agreement not only on position but also on velocity, therefore this optimized control is more challenging and interesting. To derive the control, reinforcement learning (RL) can be a natural consideration because it can overcome the difficulty of solving the Hamilton–Jacobi–Bellman (HJB) equation. To implement RL, it needs to iterate both adaptive critic and actor networks each other. However, if this optimized control learns RL from most existing optimal methods that derives the critic and actor adaptive laws from the negative gradient of square of the approximating function of the HJB equation, this control algorithm will be very intricate, because the HJB equation correlated to a second-order nonlinear multiagent system will become very complex due to strong state coupling and nonlinearity. In this work, since the two RL adaptive laws are derived via implementing the gradient descent method to a simple positive function, which is obtained on the basis of a partial derivative of the HJB equation, this optimized control is significantly simple. Meanwhile, it not merely can avoid the requirement of known dynamic acknowledge, but also can release the condition of persistent excitation, which is demanded in most RL optimization methods for training the adaptive parameter more sufficiently. Finally, the proposed control is demonstrated by both theory and computer simulation.

In this article, a control scheme based on optimized backstepping (OB) technique is developed for a class of nonlinear strict-feedback systems with unknown dynamic functions. Reinforcement learning (RL) is employed for achieving the optimized control, and it is designed on the basis of the neural-network (NN) approximations under identifier-critic-actor architecture, where the identifier, critic, and actor are utilized for estimating the unknown dynamic, evaluating the system performance, and implementing the control action, respectively. OB control is to design all virtual controls and the actual control of backstepping to be the optimized solutions of corresponding subsystems. If the control is developed by employing the existing RL-based optimal control methods, it will become very intricate because their critic and actor updating laws are derived by carrying out gradient descent algorithm to the square of Bellman residual error, which is equal to the approximation of the Hamilton-Jacobi-Bellman (HJB) equation that contains multiple nonlinear terms. In order to effectively accomplish the optimized control, a simplified RL algorithm is designed by deriving the updating laws from the negative gradient of a simple positive function, which is generated from the partial derivative of the HJB equation. Meanwhile, the design can also release the condition of persistence excitation, which is required in most existing optimal controls. Finally, effectiveness is demonstrated by both theory and simulation.

In this paper, a tracking control approach for surface vessel is developed based on the new control technique named optimized backstepping (OB), which considers optimization as a backstepping design principle. Since surface vessel systems are modeled by second-order dynamic in strict feedback form, backstepping is an ideal technique for finishing the tracking task. In the backstepping control of surface vessel, the virtual and actual controls are designed to be the optimized solutions of corresponding subsystems, therefore the overall control is optimized. In general, optimization control is designed based on the solution of Hamilton-Jacobi-Bellman equation. However, solving the equation is very difficult or even impossible due to the inherent nonlinearity and complexity. In order to overcome the difficulty, the reinforcement learning (RL) strategy of actor-critic architecture is usually considered, of which the critic and actor are utilized for evaluating the control performance and executing the control behavior, respectively. By employing the actor-critic RL algorithm for both virtual and actual controls of the vessel, it is proven that the desired optimizing and tracking performances can be arrived. Simulation results further demonstrate effectiveness of the proposed surface vessel control.

This paper proposes an optimized tracking control approach using neural network (NN) based reinforcement learning (RL) for a class of nonlinear dynamic systems, which requires both tracking and optimizing to be performed simultaneously. Generally, for obtaining optimal control solution, Hamilton-Jacobi-Bellman equation is expected to be solvable, but, owing to strong nonlinearity, the equation is solved difficultly or even impossibly by analytical methods. Therefore, adaptive NN approximation based RL is usually considered. In the optimized control design, for driving output state following to the desired trajectory, an error term is split from optimal performance index function, and then both actor and critic NNs are built to perform RL algorithm. Actor NN aims to execute control behaviors, and critic NN aims to appraise control performance and make feedback to actor. The proof of stability concludes that the desired control performances are obtained. A numerical simulation is designed and implemented, and the desired results are shown.

This paper focuses on the simulation and comparison of different types of first-order decentralized sliding mode estimators (FDSMEs). From the previous works, three types of FDSMEs were presented and applied to solve the cooperative control problems. Utilizing the FDSME, a finite-time leader-follower tracking control algorithm is proposed for a networked single-integrator vehicle system. Then based on the existing structure of FDSME, a new compound FDSME is developed to improve the estimation performance. Simulation and comparison of all the presented FDSMEs are given in detail to evaluate the theoretical results. Finally, regulation rules of the parameters in the compound FDSME are summarized according to the simulation results.

The paper proposes an optimized leader-follower formation control for the multi-agent systems with unknown nonlinear dynamics. Usually, optimal control is designed based on the solution of the Hamilton-Jacobi-Bellman equation, but it is very difficult to solve the equation because of the unknown dynamic and inherent nonlinearity. Specifically, to multi-agent systems, it will become more complicated owing to the state coupling problem in control design. In order to achieve the optimized control, the reinforcement learning algorithm of the identifier-actor-critic architecture is implemented based on fuzzy logic system (FLS) approximators. The identifier is designed for estimating the unknown multi-agent dynamics; the actor and critic FLSs are constructed for executing control behavior and evaluating control performance, respectively. According to Lyapunov stability theory, it is proven that the desired optimizing performance can be arrived. Finally, a simulation example is carried out to further demonstrate the effectiveness of the proposed control approach.

In this paper, a control technique named optimized backstepping is first proposed by implementing tracking control for a class of strict-feedback systems, which considers optimization as a design philosophy of the high-order system control. The basic idea is that designing the actual and virtual controls of backstepping is the optimized solutions of the corresponding subsystems so that overall control of the high-order system is optimized. In general, optimization control is designed based on the solution of Hamilton-Jacobi-Bellman equation, but solving the equation is very difficult due to the inherent nonlinearity and intractability. In order to overcome the difficulty, the neural network (NN)-based reinforcement learning strategy of actor-critic architecture is used. In every backstepping step, the actor and critic NNs are constructed for executing control behavior and evaluating control performance, respectively. According to the Lyapunov stability theorem, it is proven that the desired control performance can be obtained. Finally, a simulation example is carried out to further demonstrate the effectiveness of the proposed control approach.

Compared with the existing neural network (NN) or fuzzy logic system (FLS) based adaptive consensus methods, the proposed approach can greatly alleviate the computation burden because it needs only to update a few adaptive parameters online. In the multiagent agreement control, the system uncertainties derive from the unknown nonlinear dynamics are counteracted by employing the adaptive NNs; the state delays are compensated by designing a Lyapunov-Krasovskii functional. Finally, based on Lyapunov stability theory, it is demonstrated that the proposed consensus scheme can steer a multiagent system synchronizing to the predefined reference signals. Two simulation examples, a numerical multiagent system and a practical multimanipulator system, are carried out to further verify and testify the effectiveness of the proposed agreement approach.

Combined with backstepping techniques, an observer-based adaptive consensus tracking control strategy is developed for a class of high-order nonlinear multiagent systems, of which each follower agent is modeled in a semi-strict-feedback form. By constructing the neural network-based state observer for each follower, the proposed consensus control method solves the unmeasurable state problem of high-order nonlinear multiagent systems. The control algorithm can guarantee that all signals of the multiagent system are semi-globally uniformly ultimately bounded and all outputs can synchronously track a reference signal to a desired accuracy. A simulation example is carried out to further demonstrate the effectiveness of the proposed consensus control method.

This paper studies an adaptive tracking control for a class of nonlinear stochastic systems with unknown functions. The considered systems are in the nonaffine pure-feedback form, and it is the first to control this class of systems with stochastic disturbances. The fuzzy-neural networks are used to approximate unknown functions. Based on the backstepping design technique, the controllers and the adaptation laws are obtained. Compared to most of the existing stochastic systems, the proposed control algorithm has fewer adjustable parameters and thus, it can reduce online computation load. By using Lyapunov analysis, it is proven that all the signals of the closed-loop system are semiglobally uniformly ultimately bounded in probability and the system output tracks the reference signal to a bounded compact set. The simulation example is given to illustrate the effectiveness of the proposed control algorithm.

Because of the complexity of consensus control of nonlinear multiagent systems in state time-delay, most of previous works focused only on linear systems with input time-delay. An adaptive neural network (NN) consensus control method for a class of nonlinear multiagent systems with state time-delay is proposed in this paper. The approximation property of radial basis function neural networks (RBFNNs) is used to neutralize the uncertain nonlinear dynamics in agents. An appropriate Lyapunov-Krasovskii functional, which is obtained from the derivative of an appropriate Lyapunov function, is used to compensate the uncertainties of unknown time delays. It is proved that our proposed approach guarantees the convergence on the basis of Lyapunov stability theory. The simulation results of a nonlinear multiagent time-delay system and a multiple collaborative manipulators system show the effectiveness of the proposed consensus control algorithm.

This paper studies an adaptive neural consensus control for a class of nonlinear multi-agent time delay systems. The Radial Basis Function Neural Networks (RBFNN) are utilized to approximate the unknown nonlinear function of system dynamic. Based on Lyapunov analysis method, it is proven that the nonlinear multi-agent system is stable and the consensus errors converge to a small neighborhood of zero. In contrast to the existing results, the advantage of the developed scheme is that the influence of time delay on the nonlinear multi-agent systems is eliminated. The effectiveness of the developed scheme is illustrated by a simulation example.

In this paper, we described a class of nonlinear multi-agent dynamic systems in consensus problem which every agent is multi-dimensional dynamic system. The control objective is expect to find suitable consensus controller for every agent such that 1) guarantee all the signals in the dynamic systems remain bounded. 2) design the consensus controllers for every agent then the average-consensus behavior of the multi-agent systems can be obtained.

This brief studies an adaptive neural output feedback tracking control of uncertain nonlinear multi-input-multi-output (MIMO) systems in the discrete-time form. The considered MIMO systems are composed of n subsystems with the couplings of inputs and states among subsystems. In order to solve the noncausal problem and decouple the couplings, it needs to transform the systems into a predictor form. The higher order neural networks are utilized to approximate the desired controllers. By using Lyapunov analysis, it is proven that all the signals in the closed-loop system is the semi-globally uniformly ultimately bounded and the output errors converge to a compact set. In contrast to the existing results, the advantage of the scheme is that the number of the adjustable parameters is highly reduced. The effectiveness of the scheme is verified by a simulation example.