In essence, reinforcement learning (RL) solves optimal control problem (OCP) by employing a neural network (NN) to fit the optimal policy from state to action. The accuracy of policy approximation is often very low in complex control tasks, leading to unsatisfactory control performance compared with online optimal controllers. A primary reason is that the landscape of value function is always not only rugged in most areas but also flat on the bottom, which damages the convergence to the minimum point. To address this issue, we develop a bicriteria policy optimization (BPO) algorithm, which leverages a few optimal demonstration trajectories to guide the policy search at the gradient level. Different from conventional problem definition, BPO seeks to solve a bicriteria OCP, which has two homomorphic objectives: one is from the standard reward signals and the other is to align the demonstration trajectories. We introduce two co-state variables, one for each objectives, and formulate two Hamiltonians for this bicriteria OCP. The resulting new optimality condition preserves the minimum values of both Hamiltonians. Furthermore, we find that gradient conflict is a key obstacle to simultaneously descending both Hamiltonians, and its impact is negatively proportional to the inner product between the ideal and actual gradients. A minimax optimization problem is built at each RL iteration to minimize conflicts between two homomorphic objectives, whose solution for policy updating is referred to as harmonic gradient. By converting its inner optimization loop into a linear programming with convex trust region constraint, we simplify this problem into a single-loop maximization problem with much increased computational efficiency. Experiment tests on both linear and nonlinear control tasks validate the effectiveness of our BPO algorithm on the accuracy improvement of policy network.

Exploration is crucial in Reinforcement Learning (RL) as it enables the agent to understand the environment for better decision-making. Existing exploration methods fall into two paradigms: active exploration, which injects stochasticity into the policy but struggles in high-dimensional environments, and passive exploration, which manages the replay buffer to prioritize under-explored regions but lacks sample diversity. To address the limitation in passive exploration, we propose Modelic Generative Exploration (MoGE), which augments exploration through the generation of under-explored critical states and synthesis of dynamics-consistent experiences. MoGE consists of two components: (1) a diffusion generator for critical states under the guidance of entropy and TD error, and (2) a one-step imagination world model for constructing critical transitions for agent learning. Our method is simple to implement and seamlessly integrates with mainstream off-policy RL algorithms without structural modifications. Experiments on OpenAI Gym and DeepMind Control Suite demonstrate that MoGE, as an exploration augmentation, significantly enhances efficiency and performance in complex tasks.

Online planning has proven effective in reinforcement learning (RL) for improving sample efficiency and final performance. However, using planning for environment interaction inevitably introduces a divergence between the collected data and the policy's actual behaviors, degrading both model learning and policy improvement. To address this, we propose BOOM (Bootstrap Off-policy with WOrld Model), a framework that tightly integrates planning and off-policy learning through a bootstrap loop: the policy initializes the planner, and the planner refines actions to bootstrap the policy through behavior alignment. This loop is supported by a jointly learned world model, which enables the planner to simulate future trajectories and provides value targets to facilitate policy improvement. The core of BOOM is a likelihood-free alignment loss that bootstraps the policy using the planner’s non-parametric action distribution, combined with a soft value-weighted mechanism that prioritizes high-return behaviors and mitigates variability in the planner’s action quality within the replay buffer. Experiments on the high-dimensional DeepMind Control Suite and Humanoid-Bench show that BOOM achieves state-of-the-art results in both training stability and final performance. The code is accessible at \url{https://github.com/molumitu/BOOM_MBRL}.

Training deep reinforcement learning (RL) agents necessitates overcoming the highly unstable nonconvex stochastic optimization inherent in the trial-and-error mechanism. To tackle this challenge, we propose a physics-inspired optimization algorithm called relativistic adaptive gradient descent (RAD), which enhances long-term training stability. By conceptualizing neural network (NN) training as the evolution of a conformal Hamiltonian system, we present a universal framework for transferring long-term stability from conformal symplectic integrators to iterative NN updating rules, where the choice of kinetic energy governs the dynamical properties of resulting optimization algorithms. By utilizing relativistic kinetic energy, RAD incorporates principles from special relativity and limits parameter updates below a finite speed, effectively mitigating abnormal gradient influences. In addition, RAD models NN optimization as the evolution of a multiparticle system where each trainable parameter acts as an independent particle with an individual adaptive learning rate. We prove RAD's sublinear convergence under general nonconvex settings, where smaller gradient variance and larger batch sizes contribute to tighter convergence. Notably, RAD degrades to the well-known adaptive moment estimation (ADAM) algorithm when its speed coefficient is chosen as one and symplectic factor as a small positive value. Experimental results show RAD outperforming nine baseline optimizers with five RL algorithms across twelve environments, including standard benchmarks and challenging scenarios. Notably, RAD achieves up to a 155.1% performance improvement over ADAM in Atari games, showcasing its efficacy in stabilizing and accelerating RL training.

The effective design of state representation plays a pivotal role in bridging the gap between perception and downstream decision-making and control in autonomous driving systems based on reinforcement learning. Among perception observations, accurately representing the set of surrounding participants poses significant challenges due to its variable size and unordered nature. These challenges result in dimension sensitivity and permutation sensitivity issues, respectively. To systematically tackle these complexities, we introduce a general framework for learning a state representation module, which can be regarded as a combination of any transformation and aggregation modules. Specifically, we employ a transformer encoder as the transformation module to attract features individually and design a novel aggregation module called Feature-wise Sorted Query (FSQ) to effectively aggregate the feature set into a state representation vector while adaptively attending to essential participant features. In FSQ, the feature-wise sort and latent query attention mechanisms can address permutation sensitivity and dimension sensitivity, respectively. Additionally, we propose an offline training paradigm to decouple state representation learning from downstream reinforcement learning tasks, enhancing stability and avoiding overfitting for particular scenarios. Simulation and experimental results demonstrate the superior performance and a more stable training process of our method across six varying-size set regression tasks and downstream autonomous driving tasks, particularly when leveraging the offline training paradigm.