In this article, a novel accelerated evolution-guided Q-learning (EGQL) algorithm is introduced to address optimal control problems for unknown nonlinear systems. A novel adaptive evolutionary algorithm is introduced here to enhance problem-solving strategies in two key aspects, which is a concept referred to dual swarm intelligence. First, the novel evolutionary algorithm is incorporated into policy improvement, replacing the gradient descent approach in traditional Q-learning. The method eliminates the need for gradient information in the approximate Q-function, while also enabling more precise policy solutions. Second, the commonly used polynomial model is replaced with a neural network, which is further optimized by the evolutionary algorithm to approximate the critic network. The accuracy of the approximate Q-function is significantly enhanced, improving the precision of value function update. Furthermore, the monotonicity and convergence properties of the algorithm are thoroughly analyzed. Finally, the effectiveness of the EGQL method is validated through two simulation experiments, demonstrating its superiority over traditional Q-learning.

The innovation of optimal learning control methods is profoundly propelled due to the improvement of the learning ability. In this article, we investigate the synthesis of initialization and acceleration for optimal learning control algorithms. This approach contrasts with traditional methods that concentrate solely on either the improvement of initialization or acceleration. Specifically, we establish a novel relaxed policy iteration (PI) algorithm with self-learning horizon for stochastic optimal control. Notably, by suitably utilizing self-learning horizon, we can directly evaluate inadmissible policies to reduce the initialization burden. Meanwhile, the inadmissible policy can be rapidly optimized with few learning iterations. Then, several critical conclusions of relaxed optimal control are established by discussing algorithm convergence and system stability. Furthermore, to provide the convincing application potentials, a class of unconventional problems is effectively solved by the relaxed PI algorithm, including the dynamics with external noises and nonzero equilibrium. Finally, we present a series of nonlinear benchmarks with practical applications to comprehensively evaluate the performance of relaxed PI. The experimental results obtained from these diverse benchmarks uniformly highlight the effectiveness of self-learning horizon mechanism.

Data-driven methods have significantly advanced optimal learning control, but some approaches overlook systematic considerations of data utilization, including safety, efficiency, and error accumulation. To address the neglects in safe neural critic control, this article introduces a parallel multistep evaluation mechanism that combines data from the system interaction with data generated by data-driven models. Based on this evaluation mechanism, we propose a novel parallel multistep Q-learning algorithm that enhances data utilization efficiency and mitigates the error accumulation. Furthermore, we formulate a novel control barrier function (CBF) to ensure safety during learning and control processes, which is capable of dealing with asymmetric constraints and adjusting the constraint strength. In addition, the analysis reveals that multistep information introduced by data-driven models influences the learning performance of actor-critic neural networks (NNs). Finally, parallel multistep Q-learning, which makes use of data in aspects of safety, efficiency, and error bounds, is validated within an orbital maneuver system.

This article investigates critical challenges in optimal output tracking control, including residual tracking errors, potential instability caused by discount factors, and premature convergence due to inefficient termination criteria. We tackle these issues by developing a data-driven parallelQ-learning algorithm. Specifically, a utility function directly linked to system states is proposed to avoid the instability and error amplification issues in traditional discounted approaches. In addition, the algorithm uses dual lightweight controllers that use convergence properties to enhance learning efficiency. Based on dual controllers, a novel termination criterion is introduced to prevent premature convergence during the training process. Numerical simulations demonstrate that the proposed method eliminates tracking errors, accelerates convergence compared with traditional algorithms, and ensures stable convergence across diverse system dynamics.

Parallelization is widely employed to improve the exploration ability of controllers. However, it is rare to provide a lightweight scheme for reducing homogeneous policies with theoretical guarantees. This article is concerned with a novel parallel scheme for solving optimal control problems. In brief, we design a novel global indicator that inherits the theoretical guarantees of a class of iterative reinforcement learning algorithms. By generating a tentative function, the global indicator can guide and communicate with parallel controllers to accelerate the learning process. Using two typical exploration policies, the novel parallel scheme can rapidly compress the neighborhood of the optimal cost function. Besides, two parallel algorithms based on value iteration and Q-learning are established to improve the data efficiency through different extensions. Finally, two benchmark problems are presented to demonstrate the learning effectiveness of the novel parallel scheme.