Lexicographic multi-objective problems, which consist of multiple conflicting subtasks with explicit priorities, are common in real-world applications. Despite the advantages of Reinforcement Learning (RL) in single tasks, extending conventional RL methods to prioritized multiple objectives remains challenging. In particular, traditional Safe RL and Multi-Objective RL (MORL) methods have difficulty enforcing priority orderings efficiently. Therefore, Lexicographic Multi-Objective RL (LMORL) methods have been developed to address these challenges. However, existing LMORL methods either rely on heuristic threshold tuning with prior knowledge or are restricted to discrete domains. To overcome these limitations, we propose Lexicographically Projected Policy Gradient RL (LPPG-RL), a novel LMORL framework which leverages sequential gradient projections to identify feasible policy update directions, thereby enabling LPPG-RL broadly compatible with all policy gradient algorithms in continuous spaces. LPPG-RL reformulates the projection step as an optimization problem, and utilizes Dykstra's projection rather than generic solvers to deliver great speedups, especially for small- to medium-scale instances. In addition, LPPG-RL introduces Subproblem Exploration (SE) to prevent gradient vanishing, accelerate convergence and enhance stability. We provide theoretical guarantees for convergence and establish a lower bound on policy improvement. Finally, through extensive experiments in a 2D navigation environment, we demonstrate the effectiveness of LPPG-RL, showing that it outperforms existing state-of-the-art continuous LMORL methods.

In real-world industrial processes, there are many lexicographic multiobjective problems, which consist of conflicting subtasks with explicit priorities. In the past, model predictive control (MPC) is combined with lexicographic optimization to form Lexicographic MPC (LMPC), which handles priorities in a hierarchical manner. Although higher priorities like safety in chemical reactions, can be strictly maintained, the recursive optimization incurs great burden for online computation, which makes LMPC impractical. Considering the great success, especially the low online computation cost of reinforcement learning (RL), several studies have been conducted in lexicographic multiobjective RL (LMORL) to train end-to-end policies for prioritized problems. However, existing LMORL methods are either limited to discrete domains with low precision control in processes, or heavily reliant on hyperparameter tuning with prior knowledge. To address these limitations in continuous process problems, lexicographically projected policy gradient RL (LPPG-RL) is proposed. LPPG-RL performs sequential policy gradient projections to identify a feasible policy update direction, ensuring compatibility with general continuous policy gradient RL algorithms. LPPG-RL employs Dykstra's projection instead of generic solvers to accelerate the direction search, particularly for high-dimension gradient vectors. In addition, LPPG-RL introduces subproblem exploration to mitigate gradient vanishing and improve training stability. Furthermore, convergence proof and stepsize analysis are provided. Finally, two CSTR problems serve as the case study, demonstrating the effectiveness of LPPG-RL through comparisons with baseline LMORL and LMPC methods.

At present, working conditions are becoming more complex, and operation policy requirements are more diverse in process system engineering. To control a process problem, a balance must be found between speed and stability, in that operations should sometimes be faster and other times smoother. Traditional controllers, such as PID and model predictive control are applied in various problems, and some parameters in controllers can be used to represent the operation policy. However, there can be difficulties in tuning parameters, and time costs of online calculation. This article proposes parametric deep reinforcement learning (PDRL) to replace traditional controllers. PDRL has two parts. A vanilla DRL framework is adapted to solve the setpoint tracking problem. With a state and a reward function and robust training tricks, trained agents can be applied to more generalized working conditions. Base agents of different operation policies are trained in advance. With target performance from operators, the target policy can be fitted by base agents with a set of weights, which are first optimized by minimizing the squared error between the target and fitted policy in a basic task, and applied to generalized conditions. A shell benchmark problem is chosen as a case study, whose results show that PDRL has feasibility and stability both in basic and generalized tasks, even in a noisy environment.

At an urban intersection, traffic performance is highly influenced not only by signal timing schemes but also by lane assignments. However, dynamic adjustment of lane configurations remains understudied due to the prevalent use of predetermined lane arrangements. To address this research gap, we propose the TransLane framework, which collaboratively optimizes time and space resources within an intersection by implementing dynamic lanes. We model each intersection as an agent and apply a hierarchical recommendation approach to these agents. Additionally, the framework addresses the challenge of training an agent in the absence of sufficient data by using transfer learning techniques. Specifically, a new intersection agent can be trained from pre-trained agents via model and sample transfer mechanisms, which facilitate the reuse of traffic control knowledge from similar intersections. The efficacy and superiority of TransLane are demonstrated through simulation studies that show it outperforms models that optimize space or time resources independently using widely adopted traffic control optimization methods.

The advanced modular high-temperature gas-cooled reactor (MHTGR) power plant is a novel nuclear power system developed in China. To ensure load match and system safety of the MHTGR power plant, this paper investigates operational feasibility under uncertainty by introducing a concept of probabilistic operating space (POS), which is characterized by the envelope of operations that provides assurance of control performance with desired reliability. A model-based approach is proposed to efficiently identify the POS, considering hybrid uncertainties of process variability and model mismatch. First, Sobol sensitivity analysis and parameter estimation are applied for uncertainty quantification. Second, the hybrid uncertainties are propagated by a polynomial chaos expansion surrogate with parameterized coefficients. Furthermore, nested sampling is employed to allocate operating samples towards regions of high feasibility probability. The resulting POS is finally visualized as a probability map using geometric projection. Simulation results not only validate the effectiveness of the POS approach in identifying reliable load match operations but also highlight its computational efficiency.

This paper summarizes the autonomous guidance methods (AGMs) for pinpoint soft landing on celestial surfaces. We first review the development of powered descent guidance methods, focusing on their contributions for dealing with constraints and enhancing computational efficiency. With the increasing demand for reusable launchers and more scientific returns from space exploration, pinpoint soft landing has become a basic requirement. Unlike the kilometer-level precision for previous activities, the position accuracy of future planetary landers is within tens of meters of a target respecting all constraints of velocity and attitude, which is a very difficult task and arouses renewed interest in AGMs. This paper states the generalized three- and six-degree-of-freedom optimization problems in the powered descent phase and compares the features of three typical scenarios, i.e., the lunar, Mars, and Earth landing. On this basis, the paper details the characteristics and adaptability of AGMs by comparing aspects of analytical guidance methods, numerical optimization algorithms, and learning-based methods, and discusses the convexification treatment and solution strategies for non-convex problems. Three key issues related to AGM application, including physical feasibility, model accuracy, and real-time performance, are presented afterward for discussion. Many space organizations, such as those in the United States, China, France, Germany, and Japan, have also developed free-flying demonstrators to carry out related research. The guidance methods which have been tested on these demonstrators are briefly introduced at the end of the paper.

This work formulates the multi-vehicle lane change motion planning task as a centralized optimal control problem, which is beneficial in being generic and complete. However, a direct solution to this optimal control problem is numerically intractable due to the dimensionality of the collision-avoidance constraints and nonlinearity of the vehicle kinematics. A progressively constrained dynamic optimization (PCDO) method is proposed to facilitate the numerical solving process of this complicated problem. PCDO guarantees to efficiently obtain an optimum to the original optimal control problem via solving a sequence of simplified problems which gradually judge and reserve only the active collision-avoidance constraints. A first-regularization-then-action strategy, together with the look-up table technique, is developed for online solutions. At the regularization stage, the vehicles form a standard formation by linear acceleration/deceleration only. At the action stage, the vehicles execute lane change motions computed offline and recorded in the look-up table. This makes online motion planning feasible because 1) the computational complexity at the regularization stage scales linearly rather than exponentially with the vehicle number; and 2) online computation at the action stage is fully avoided through data extraction from the look-up table.

Autonomous parking has been a widely developed branch of intelligent transportation systems. In autonomous parking, maneuver planning is a crucial procedure that determines how intelligent the entire parking system is. This paper concerns planning time-optimal parallel parking maneuvers in a straightforward, accurate, and purely objective way. A unified dynamic optimization framework is established, which includes the vehicle kinematics, physical restrictions, collision-avoidance constraints, and an optimization objective. Interior-point method (IPM)-based simultaneous dynamic optimization methodology is adopted to solve the formulated dynamic optimization problem numerically. Given that near-feasible solutions have been widely acknowledged to ease optimizing nonlinear programs (NLPs), a critical region-based initialization strategy is proposed to facilitate the offline NLP-solving process, a lookup table-based strategy is proposed to guarantee the on-site planning performance, and a receding-horizon optimization framework is proposed for online maneuver planning. A series of parallel parking cases is tested, and simulation results demonstrate that our proposal is efficient even when the slot length is merely 10.19% larger than the car length. As a unified maneuver planner, our adopted IPM-based simultaneous dynamic optimization method can deal with any user-specified demand provided that it can be explicitly described.

Trajectory planning is a critical aspect of autonomous tractor-trailer vehicle design. Trajectory planning algorithms usually compute paths first, trajectories are obtained thereafter. This multi-step feature makes those planners inefficacious to handle time-dependent constraints. In this study, we consider the original trajectory planning mission directly, which is described as an optimal control problem containing the kinematics, mechanical/physical constraints, environmental requirements as well as an optimization criterion. In this formulation, only the fundamental driving principles with no special issues (e.g., backing-up maneuver and jackknife) are considered. For example, the prevailing “small-angle assumption” is not utilized to prevent jackknifing. Instead, we only require that different parts of a tractor-trailer vehicle should not collide, since the emergence of jackknife does not physically violate the kinematics. An interior-point method based simultaneous approach is adopted to solve the formulated optimal control problem. Simulation results verify our proposal is capable of handling scenarios with various user-specified requirements.

This paper proposes a motion planner for autonomous parking. Compared to the prevailing and emerging studies that handle specific or regular parking scenarios only, our method describes various kinds of parking cases in a unified way regardless they are regular parking scenarios (e.g., parallel, perpendicular or echelon parking cases) or not. First, we formulate a time-optimal dynamic optimization problem with vehicle kinematics, collision-avoidance conditions and mechanical constraints strictly described. Thereafter, an interior-point simultaneous approach is introduced to solve that formulated dynamic optimization problem. Simulation results validate that our proposed motion planning method can tackle general parking scenarios. The tested parking scenarios in this paper can be regarded as benchmark cases to evaluate the efficiency of methods that may emerge in the future. Our established dynamic optimization problem is an open and unified framework, where other complicated user-specific constraints/optimization criteria can be handled without additional difficulty, provided that they are expressed through inequalities/polynomial explicitly. This proposed motion planner may be suitable for the next-generation intelligent parking-garage system.

We propose a novel kind of termination criteria, reduced precision solution (RPS) criteria, for solving optimal control problems (OCPs) in nonlinear model predictive control (NMPC), which should be solved quickly for new inputs to be applied in time. Computational delay, which may destroy the closed-loop stability, usually arises while non-convex and nonlinear OCPs are solved with differential equations as the constraints. Traditional termination criteria of optimization algorithms usually involve slow convergence in the solution procedure and waste computing resources. Considering the practical demand of solution precision, RPS criteria are developed to obtain good approximate solutions with less computational cost. These include some indices to judge the degree of convergence during the optimization procedure and can stop iterating in a timely way when there is no apparent improvement of the solution. To guarantee the feasibility of iterate for the solution procedure to be terminated early, the feasibility-perturbed sequential quadratic programming (FP-SQP) algorithm is used. Simulations on the reference tracking performance of a continuously stirred tank reactor (CSTR) show that the RPS criteria efficiently reduce computation time and the adverse effect of computational delay on closed-loop stability.

A model predictive controller was designed in this study for a single supply chain unit. A demand model was described using an autoregressive integrated moving average (ARIMA) model, one that is identified on-line to forecast the future demand. Feedback was used to modify the demand prediction, and profit was chosen as the control objective. To imitate reality, the purchase price was assumed to be a piecewise linear form, whereby the control objective became a nonlinear problem. In addition, a genetic algorithm was introduced to solve the problem. Constraints were put on the predictive inventory to control the inventory fluctuation, that is, the bullwhip effect was controllable. The model predictive control (MPC) method was compared with the order-up-to-level (OUL) method in simulations. The results revealed that using the MPC method can result in more profit and make the bullwhip effect controllable.