Learning to perform procedural motion or manipulation tasks in unstructured or uncertain environments poses significant challenges for intelligent agents. Although reinforcement learning algorithms have demonstrated positive results on simple tasks, the hard-to-engineer reward functions and the impractical amount of trial-and-error iterations these agents require in long-experience streams still present challenges for deployment in industrially relevant environments. In this regard, interactive reinforcement learning has emerged as a promising approach to mitigate these limitations, whereby a human supervisor provides evaluative or corrective feedback to the learning agent during training. However, the requirement of a human-in-the-loop approach throughout the learning process can be impractical for tasks that span several hours. This study aims to overcome this limitation by automating the learning process and substituting human feedback with an artificial supervisor grounded in constraint-based modeling techniques. In contrast to the logical constraints commonly used for conventional reinforcement learning, constraint-based modeling techniques offer enhanced adaptability in terms of conceptualizing and modeling the human knowledge of a task. This modeling capability allows an automated supervisor to acquire a closer approximation to human reasoning by dividing complex tasks into more manageable components and identifying the associated subtask and contextual cues in which the agent is involved. The supervisor then adjusts the evaluative and corrective feedback to suit the specific subtask under consideration. The framework was assessed using three actor-critic agents in a human–robot interaction environment, demonstrating a sample efficiency improvement of 50% and success rates of ≥ 95% in simulation and 90% in real-world implementation.

Effective motion planning is an indispensable prerequisite for the optimal performance of robotic manipulators in any task. In this regard, the research and application of reinforcement learning in robotic manipulators for motion planning have gained great relevance in recent years. The ability of reinforcement learning agents to adapt to variable environments, especially those featuring dynamic obstacles, has propelled their increasing application in this domain. Notwithstanding, a clear need remains for a resource that critically examines the progress, challenges, and future directions of this machine learning control technique in motion planning. This article undertakes a comprehensive review of the landscape of reinforcement learning, offering a retrospective analysis of its application in motion planning from 2018 to the present. The exploration extends to the trends associated with reinforcement learning in the context of serial manipulators and motion planning, as well as the various technological challenges currently presented by this machine learning control technique. The overarching objective of this review is to serve as a valuable resource for the robotics community, facilitating the ongoing development of systems controlled by reinforcement learning. By delving into the primary challenges intrinsic to this technology, the review seeks to enhance the understanding of reinforcement learning’s role in motion planning and provides insights that may suggest future research directions in this domain.

To enable flexible quality inspection in industrial manufacturing environments, there is an increasing demand for easy-to-use automation systems that can aid factory workers in repetitive tasks. However, quality-control tasks may be too complex for a single classification algorithm to deliver results comparable to a human operator. Therefore, we propose a computer-vision architecture to improve flexibility of quality inspection tasks, that first detects objects of interest, estimates their poses and defines regions-of-interest prior to the classification task takes place. The proposed hybrid architecture consists of a deep-learning based object detection and segmentation model, and edge-based pose estimation method. We validate our approaches in texture-less, reflecting and symmetric vehicle metal sheets which are challenging for state-of-the-art object detection and pose estimation methods. Furthermore, since annotating the data required by these methods is laborious and time-consuming, we train our object detection and pose estimation architectures on 3D synthetic datasets based on available CAD model. We demonstrate promising domain generalization results on the object detection stage of our architecture (Mask R-CNN) as while trained with synthetic data, it reaches a bounding box mAP of 0.781 on synthetic and mAP of 0.686 on real images. On the other side, our simple edge-based pose estimation method can cope with texture-less parts even if using synthetic data as reference and being evaluated on real images. Our custom edge-based pose estimator reaches 16° of average rotation error and 0.14m of average translation error on real images.

skrl is an open-source modular library for reinforcement learning written in Python and designed with a focus on readability, simplicity, and transparency of algorithm implementations. In addition to supporting environments that use the traditional interfaces from OpenAI Gym/Farama Gymnasium, DeepMind and others, it provides the facility to load, configure, and operate NVIDIA Isaac Gym, Isaac Orbit, and Omniverse Isaac Gym environments. Furthermore, it enables the simultaneous training of several agents with customizable scopes (subsets of environments among all available ones), which may or may not share resources, in the same run. The library's documentation can be found at https://skrl.readthedocs.io and its source code is available on GitHub at https://github.com/Toni-SM/skrl.

We present a deep reinforcement learning-based approach to control robotic manipulators and construct task-independent trajectories for point-to-point motions. The research objective in this work is to learn control in the joint action space, which can be generalized to various industrial manipulators. The approach necessitates that the neural network learns a mapping from joint movements to the reward landscape determined by the distance to the goal and nearby obstacles. In addition, curriculum learning is embedded in this approach to facilitate learning by reducing the complexity of the environment. Conducted experiments demonstrate how the reinforcement learning-based approach can be applied to three different industrial manipulators in simulation with minimal configuration changes. The results of our contribution demonstrate that a model can be trained in a simulation environment, transferred to the real world, and used in complex environments. Furthermore, the Sim2Real transfer, augmented by curriculum learning, highlights that the robots behave in the same way in the real world as in the simulation and that the operations in the real world are safe from a control and trajectory point-of-view.

Extraterrestrial rovers with a general-purpose robotic arm have many potential applications in lunar and planetary exploration. Introducing autonomy into such systems is desirable for increasing the time that rovers can spend gathering scientific data and collecting samples. This work investigates the applicability of deep reinforcement learning for vision-based robotic grasping of objects on the Moon. A novel simulation environment with procedurally-generated datasets is created to train agents under challenging conditions in unstructured scenes with uneven terrain and harsh illumination. A model-free off-policy actor-critic algorithm is then employed for end-to-end learning of a policy that directly maps compact octree observations to continuous actions in Cartesian space. Experimental evaluation indicates that 3D data representations enable more effective learning of manipulation skills when compared to traditionally used image-based observations. Domain randomization improves the generalization of learned policies to novel scenes with previously unseen objects and different illumination conditions. To this end, we demonstrate zero-shot sim-to-real transfer by evaluating trained agents on a real robot in a Moon-analogue facility. The source code and datasets are available at https://github.com/AndrejOrsula/drl_grasping.

In the rising area of close human-robot collaboration in industrial scenarios, the human operator must be able to easily understand the intent of and data from the robot. Shipbuilding environments exhibit unique features, which make deployment of mobile robots both challenging, relevant, and interesting. One task that is still solely carried out manually today due to its complexity and high need for mobility is the fit-out operation stud welding. This paper presents the latest state-of-the-art developments in human-robot interaction (HRI) for robotic stud welding in large semi-structured manufacturing spaces. The welding itself is carried out autonomously by an autonomous industrial mobile manipulator (AIMM). A novel HRI is proposed, which employs projection mapping and an IMU device to enable intuitive and natural interaction with the robot. Task specific information is projected directly into task space as augmented reality using a projector mounted on the robot end-effector. The IMU device enables non-expert operators to program, verify, and reprogram the robot's task on-site in a ship superstructure. The usability of the system is tested in an extensive user test. It is concluded that non-experts after a short introduction are able to both modify a previous task and instruct and a new task using on average 1:01 and 1:16 minutes. Finally, the HRI has been implemented on a prototype robot and tested in an actual shipyard facility. The precision of the system, including operator inaccuracy, was evaluated to have a standard deviation of 3.6mm.

Increased customer needs and intensified global competition require intelligent and flexible automation. The interaction technology mobile robotics addresses this, so it holds great potential within the industry. This paper presents the concepts, ideas and working principles of the mobile robot "Little Helper" - an ongoing research project at Aalborg University, Denmark, concerning the development of an autonomous and flexible manufacturing assistant. To demonstrate the "Little Helper" concept a full-scale prototype has been built and experiments carried out. Experiences and knowledge gained from this show promising results regarding industrial integration, exploitation and maturation of mobile robotics.