Lexicographic multi-objective problems, which consist of multiple conflicting subtasks with explicit priorities, are common in real-world applications. Despite the advantages of Reinforcement Learning (RL) in single tasks, extending conventional RL methods to prioritized multiple objectives remains challenging. In particular, traditional Safe RL and Multi-Objective RL (MORL) methods have difficulty enforcing priority orderings efficiently. Therefore, Lexicographic Multi-Objective RL (LMORL) methods have been developed to address these challenges. However, existing LMORL methods either rely on heuristic threshold tuning with prior knowledge or are restricted to discrete domains. To overcome these limitations, we propose Lexicographically Projected Policy Gradient RL (LPPG-RL), a novel LMORL framework which leverages sequential gradient projections to identify feasible policy update directions, thereby enabling LPPG-RL broadly compatible with all policy gradient algorithms in continuous spaces. LPPG-RL reformulates the projection step as an optimization problem, and utilizes Dykstra's projection rather than generic solvers to deliver great speedups, especially for small- to medium-scale instances. In addition, LPPG-RL introduces Subproblem Exploration (SE) to prevent gradient vanishing, accelerate convergence and enhance stability. We provide theoretical guarantees for convergence and establish a lower bound on policy improvement. Finally, through extensive experiments in a 2D navigation environment, we demonstrate the effectiveness of LPPG-RL, showing that it outperforms existing state-of-the-art continuous LMORL methods.

In real-world industrial processes, there are many lexicographic multiobjective problems, which consist of conflicting subtasks with explicit priorities. In the past, model predictive control (MPC) is combined with lexicographic optimization to form Lexicographic MPC (LMPC), which handles priorities in a hierarchical manner. Although higher priorities like safety in chemical reactions, can be strictly maintained, the recursive optimization incurs great burden for online computation, which makes LMPC impractical. Considering the great success, especially the low online computation cost of reinforcement learning (RL), several studies have been conducted in lexicographic multiobjective RL (LMORL) to train end-to-end policies for prioritized problems. However, existing LMORL methods are either limited to discrete domains with low precision control in processes, or heavily reliant on hyperparameter tuning with prior knowledge. To address these limitations in continuous process problems, lexicographically projected policy gradient RL (LPPG-RL) is proposed. LPPG-RL performs sequential policy gradient projections to identify a feasible policy update direction, ensuring compatibility with general continuous policy gradient RL algorithms. LPPG-RL employs Dykstra's projection instead of generic solvers to accelerate the direction search, particularly for high-dimension gradient vectors. In addition, LPPG-RL introduces subproblem exploration to mitigate gradient vanishing and improve training stability. Furthermore, convergence proof and stepsize analysis are provided. Finally, two CSTR problems serve as the case study, demonstrating the effectiveness of LPPG-RL through comparisons with baseline LMORL and LMPC methods.

At present, working conditions are becoming more complex, and operation policy requirements are more diverse in process system engineering. To control a process problem, a balance must be found between speed and stability, in that operations should sometimes be faster and other times smoother. Traditional controllers, such as PID and model predictive control are applied in various problems, and some parameters in controllers can be used to represent the operation policy. However, there can be difficulties in tuning parameters, and time costs of online calculation. This article proposes parametric deep reinforcement learning (PDRL) to replace traditional controllers. PDRL has two parts. A vanilla DRL framework is adapted to solve the setpoint tracking problem. With a state and a reward function and robust training tricks, trained agents can be applied to more generalized working conditions. Base agents of different operation policies are trained in advance. With target performance from operators, the target policy can be fitted by base agents with a set of weights, which are first optimized by minimizing the squared error between the target and fitted policy in a basic task, and applied to generalized conditions. A shell benchmark problem is chosen as a case study, whose results show that PDRL has feasibility and stability both in basic and generalized tasks, even in a noisy environment.