Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: *Can we achieve straightforward yet effective offline and online learning without introducing extra conservatism or regularization?* In this study, we propose Uni-O4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent can transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-O4 leverages diverse ensemble policies to address the mismatch issues between the estimated behavior policy and the offline dataset. Through a simple offline policy evaluation (OPE) approach, Uni-O4 can achieve multi-step policy improvement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization as well as stable and rapid online fine-tuning capabilities. Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously unseen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning. [Our website](uni-o4.github.io)

Quadruped robots are progressively being integrated into human environments. Despite the growing locomotion capabilities of quadrupedal robots, their interaction with objects in realistic scenes is still limited. While additional robotic arms on quadrupedal robots enable manipulating objects, they are sometimes redundant given that a quadruped robot is essentially a mobile unit equipped with four limbs, each possessing 3 degrees of freedom (DoFs). Hence, we aim to empower a quadruped robot to execute real-world manipulation tasks using only its legs. We decompose the loco-manipulation process into a low-level reinforcement learning (RL)-based controller and a high-level Behavior Cloning (BC)-based planner. By parameterizing the manipulation trajectory, we synchronize the efforts of the upper and lower layers, thereby leveraging the advantages of both RL and BC. Our approach is validated through simulations and real-world experiments, demonstrating the robot’s ability to perform tasks that demand mobility and high precision, such as lifting a basket from the ground while moving, closing a dishwasher, pressing a button, and pushing a door.

As the intelligent manufacturing paradigm evolves, it is urgent to design a near real-time decision-making framework for handling the uncertainty and complexity of production line control. The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the manufacturing industry. However, it is still challenging to obtain high-quality schedules for DFJSP with dynamic job arrivals in real-time, especially facing thousands of operations from a large-scale scene with complex contexts in an assembly plant. This article aims to propose a novel end-to-end hierarchical reinforcement learning framework for solving the large-scale DFJSP in near real-time. In the DFJSP, the processing information of newly arrived jobs is unknown in advance. Besides, two optimization tasks, including job operation selection and job-to-machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. In our framework, a higher-level layer is designed to automatically divide the DFJSP into subproblems, i.e., static FJSPs with different scales. And two lower-level layers are constructed to solve the subproblems. In particular, one layer based on a graph neural network is in charge of sequencing job operations, and another layer based on a multilayer perceptron is used to assign a machine to process the job operations. Numerical experiments, including offline training and online testing, are conducted on several instances with different scales. The results verify the superior performance of the proposed framework compared with existing dynamic scheduling methods, such as well-known dispatching rules and metaheuristics.

The dynamic flexible job shop scheduling problem (DFJSP) is frequently encountered in the modern manufacturing industry. As the intelligent manufacturing paradigm evolves, it is urgent to design a dynamic scheduling framework for handling the uncertainty and complexity in the real-time production line control. DFJSP aims to dynamically address new job random arrival so that the desired objective could be optimized, such as the minimization of makespan. The intractability of this problem can be directly reflected by the following two points. 1) the arrival times of new jobs are unknown in advance, so the scheduling framework needs to schedule them in real-time optimization; 2) Two optimization tasks, including job operation selection and machine assignment, have to be handled, which means multiple actions must be controlled simultaneously. This paper proposes a novel end-to-end hierarchical reinforcement learning framework to cope with the large-scale DFJSP. For generality, a higher-level layer is designed to automatically divide the DFJSP into a series of sub-problems with different scales, i.e., static FJSPs, which aims to achieve global optimization. Moreover, two lower-level layers are constructed to efficiently solve the sub-problems generated from the higher-level layer. One layer's policy based on the graph neural network is trained to schedule a job operation, and another policy based on the multi-layer perceptron is trained to assign a machine to process the job operation. Especially a Markov decision process (MDP), including state, action, and reward function, is designed for each layer of the hierarchy. Numerical experiments, including offline training and online testing, are conducted on several large-scale instances with diverse production configurations. The results verify the effectiveness of the proposed framework compared with the existing dynamic scheduling methods such as well-known dispatching rules and the existing heuristics.