Reinforcement learning (RL) algorithms often face challenges in efficiently learning effective policies for sparse-reward multi-goal robot manipulation tasks, thus requiring a vast amount of experiences. The state-of-the-art algorithm in the field, Hindsight Experience Replay (HER), addresses this issue by using failed trajectories and replacing the desired goal with hindsight goals. However, HER performs poorly when the desired goal is distant from the initial state. To address this limitation, Hindsight Goal Generation (HGG) has been proposed, which generates a curriculum of goals from already visited states. This curriculum generation is based on a single objective, and does not take obstacles into account. Here, we make a step forward by proposing Multi-Objective Evolutionary Hindsight Experience Replay (MOEHER), a novel curriculum RL algorithm that reformulates curriculum generation considering multiple objectives and obstacles. MOEHER utilizes NSGA-II to generate a curriculum that is optimized w.r.t. four objectives, namely the Q-function, the goal-proximity function, and two distance metrics, while simultaneously satisfying constraints on the obstacles. We evaluate MOEHER on four different sparse-reward robot manipulation tasks, with and without obstacles, and compare it with HER and HGG. The results demonstrate that MOEHER surpasses or performs on par with these methods on the tested tasks.

Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization (CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare it with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force/.

Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in a sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge. This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent is to achieving its goal, our method uniquely incorporates a $Q$-function and a trainable reward function based on Adversarial Intrinsic Motivation within the diffusion model. Furthermore, it promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic. This combination allows for the generation of challenging yet achievable goals, enabling agents to learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL in three different maze environments and two robotic manipulation tasks simulated in MuJoCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.