Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks. The videos are available at https://sunlighted.github.io/fosp_web/.

Offline goal-conditioned reinforcement learning (GCRL) aims at solving goal-reaching tasks with sparse rewards from an offline dataset. While prior work has demonstrated various approaches for agents to learn near-optimal policies, these methods encounter limitations when dealing with diverse constraints in complex environments, such as safety constraints. Some of these approaches prioritize goal attainment without considering safety, while others excessively focus on safety at the expense of training efficiency. In this paper, we study the problem of constrained offline GCRL and propose a new method called Recovery-based Supervised Learning (RbSL) to accomplish safety-critical tasks with various goals. To evaluate the method performance, we build a benchmark based on the robot-fetching environment with a randomly positioned obstacle and use expert or random policies to generate an offline dataset. We compare RbSL with three offline GCRL algorithms and one offline safe RL algorithm. As a result, our method outperforms the existing state-of-the-art methods to a large extent. Furthermore, we validate the practicality and effectiveness of RbSL by deploying it on a real Panda manipulator. Code is available at https://github.com/Sunlighted/RbSL.git.

In this article, we focus on human-to-cobot dual-arm handover operations for large box-type objects. The efficiency of handover operations should be ensured and the naturalness as if the handover is going on between two humans. First of all, we study the human-human dual-arm large box-type object natural handover process to guide this research. Then, for efficiency, we combine the probabilistic approach with the online learning algorithm to predict the beginning of the handover task and handover positions. The online updating probabilistic models can deal with not only human givers' regular motion patterns but also their irregular motion patterns. Then, to guarantee that human givers can perform handover operations naturally, we apply the probabilistic robot skill learning method kernelized movement primitives (KMPs) to adapt the learned receiving skills and fulfill some constraints for safety based on online predicted results. Furthermore, we give special attention to the dual-arm grasp strategy and control design to guarantee a stable grasp. In addition, we equip this handover system on a Baxter cobot and extend its grippers to make it more suitable for dual-arm handover operations. The experimental results show that the proposed handover system can solve human-to-cobot dual-arm handover operations for large box-type objects naturally and efficiently.

Learning a risk-aware policy is essential but rather challenging in unstructured robotic tasks. Safe reinforcement learning methods open up new possibilities to tackle this problem. However, the conservative policy updates make it intractable to achieve sufficient exploration and desirable performance in complex, sample-expensive environments. In this paper, we propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. Concretely, the baseline agent is responsible for maximizing rewards under standard RL settings. Thus, it is compatible with off-the-shelf training techniques of unconstrained optimization, exploration and exploitation. On the other hand, the safe agent mimics the baseline agent for policy improvement and learns to fulfill safety constraints via off-policy RL tuning. In contrast to training from scratch, safe policy correction requires significantly fewer interactions to obtain a near-optimal policy. The dual policies can be optimized synchronously via a shared replay buffer, or leveraging the pre-trained model or the non-learning-based controller as a fixed baseline agent. Experimental results show that our approach can learn feasible skills without prior knowledge as well as deriving risk-averse counterparts from pre-trained unsafe policies. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks with respect to both safety constraint satisfaction and sample efficiency.

Object handover is one of the basic abilities for the robot to interact with the human. Most of the previous works only focus on the limited handover scenarios where the robot uses one hand to give small objects to the human. In this article, we design a bidirectional bimanual handover system that enables the robot to both give and receive the big planar object with vertical grasp posture. In addition to the basic object handover function, the designed handover system also integrates a position adjustment mechanism to improve the human experience. According to different task states, the system is divided into four modes. In each mode, the robot performs a subtask and switches to the next mode at an appropriate time. We propose a two-finger grip force controller and a dual-arm admittance NN controller to control the robot to generate actual motions. By applying specific locating, trajectory planning, and signal identifying methods, we implement the designed handover system on a Baxter robot. The system is tested on two wooden plates with different widths, thicknesses, and weights. The results show that the robot can perform the handover task safely and effectively with the designed handover system.Note to Practitioners—This article aims to solve the limitation that the robot can only hand over small objects with one hand in the human–robot handover systems. In daily life, especially in carrying tasks, many objects, such as windows, wooden boards, and big frames, may also be handed over to each other. These objects can be classified as big planar objects. To enable the robot to hand over this kind of object with the human, we design a bidirectional bimanual human–robot handover system. The designed system has three main functions. First, the robot can receive the big planar object from the human and hold the object safely with two hands, which is impossible in a single-hand handover system because the weight and size of the object are large. Then, the robot can help the human hold the object or transport it to other places. Second, the robot can adjust the object handover position according to the human’s intentions while holding the object. Because the size of objects and the height of humans may different, or some tasks require that the human position be higher or lower than the robot, the current object holding position may be hard for the current human operator to take over the object. With this function, the human can move the object to an appropriate position and then take it comfortably. Third, the robot releases the object only when it gets a clear signal. Before that, the robot always grasps the object safely, and the human can freely adjust his posture. With the designed handover system, the robot can cooperate with the human to complete more tasks indoors or in factories.

In this paper, the complex problems of internal forces and position control are studied simultaneously and a disturbance observer-based radial basis function neural network (RBFNN) control scheme is proposed to: 1) estimate the unknown parameters accurately; 2) approximate the disturbance experienced by the system due to input saturation; and 3) simultaneously improve the robustness of the system. More specifically, the proposed scheme utilizes disturbance observers, neural network (NN) collaborative control with an adaptive law, and full state feedback. Utilizing Lyapunov stability principles, it is shown that semiglobally uniformly bounded stability is guaranteed for all controlled signals of the closed-loop system. The effectiveness of the proposed controller as predicted by the theoretical analysis is verified by comparative experimental studies.

Nowadays, the control technology of the robotic manipulator with flexible joints (RMFJ) is not mature enough. The flexible-joint manipulator dynamic system possesses many uncertainties, which brings a great challenge to the controller design. This paper is motivated by this problem. In order to deal with this and enhance the system robustness, the full-state feedback neural network (NN) control is proposed. Moreover, output constraints of the RMFJ are achieved, which improve the security of the robot. Through the Lyapunov stability analysis, we identify that the proposed controller can guarantee not only the stability of flexible-joint manipulator system but also the boundedness of system state variables by choosing appropriate control gains. Then, we make some necessary simulation experiments to verify the rationality of our controllers. Finally, a series of control experiments are conducted on the Baxter. By comparing with the proportional-derivative control and the NN control with the rigid manipulator model, the feasibility and the effectiveness of NN control based on flexible-joint manipulator model are verified.

The research of this paper works out the attitude and position control of the flapping wing micro aerial vehicle (FWMAV). Neural network control with full state and output feedback are designed to deal with uncertainties in this complex nonlinear FWMAV dynamic system and enhance the system robustness. Meanwhile, we design disturbance observers which are exerted into the FWMAV system via feedforward loops to counteract the bad influence of disturbances. Then, a Lyapunov function is proposed to prove the closed-loop system stability and the semi-global uniform ultimate boundedness of all state variables. Finally, a series of simulation results indicate that proposed controllers can track desired trajectories well via selecting appropriate control gains. And the designed controllers possess potential applications in FWMAVs.