Mapless navigation for Automated Guided Vehicles (AGV) via Deep Reinforcement Learning (DRL) algorithms has attracted significantly rising attention in recent years. Collision avoidance from dynamic obstacles in unstructured environments, such as pedestrians and other vehicles, is one of the key challenges for mapless navigation. Autonomous navigation requires a policy to make decisions to optimize the path distance towards the goal but also to reduce the probability of collisions with obstacles. Mostly, the reward for AGV navigation is calculated by combining multiple reward functions for different purposes, such as encouraging the robot to move towards the goal or avoiding collisions, as a state-conditioned function. The combined reward, however, may lead to biased behaviours due to the empirically chosen weights when multiple rewards are combined and dangerous situations are misjudged. Therefore, this paper proposes a learning-based method with multiple unrelated rewards, which represent the evaluation of different behaviours respectively. The policy network, named Multi-Feature Policy Gradients (MFPG), is conducted by two separate Q networks that are constructed by two individual rewards, corresponding to goal distance shortening and collision avoidance, respectively. In addition, we also propose an auto-tuning method, named Ada-MFPG, that allows the MFPG algorithm to automatically adjust the weights for the two separate policy gradients. For collision avoidance, we present a new social norm-oriented continuous biased reward for performing specific social norm so as to reduce the probabilities of AGV collisions. By adding an offset gain to one of the reward functions, vehicles conducted by the proposed algorithm exhibited the predetermined features. The work was tested in different simulation environments under multiple scenarios with a single robot or multiple robots. The proposed MFPG method is compared with standard Deep Deterministic Policy Gradient (DDPG), the modified DDPG, SAC and TD3 with a social norm mechanism. MFPG significantly increases the success rate in robot navigation tasks compared with the DDPG. Besides, among all the benchmarking algorithms, the MFPG-based algorithms have the optimal task completion duration and lower variance compared with the baselines. The work has also been tested on real robots. Experiments on the real robots demonstrate the viability of the trained model for the real world scenarios. The learned model can be used for multi-robot mapless navigation in complex environments, such as a warehouse, that need multi-robot cooperation. Our source code and supplementary material is available athttps://github.com/dornenkrone/MFPGNote to Practitioners—Autonomous navigation for AGVs in complex and large-scale environments, such as factories and warehouses, is challenging. AGVs are usually centrally controlled and depend on reliable communications. However, centralized control is not always reliable due to poor signal strengths or crashes of the server, and hence unsuitable due to the requirements of accurate information of the dynamic environments and fast responses of decision making. Therefore, it is necessary for the vehicles to perform reliable decision making based on only onboard sensors and processors, for efficient and safe autonomous navigation. Existing methods, such as simultaneous localization and mapping (SLAM) and motion planning algorithms, have been widely used. However, they are neither flexible nor generalizable enough. This paper proposes a method for autonomous navigation based on reinforcement learning (RL), which allows vehicles to gain experience through cumulative rewards by continuously interacting with the environment. The RL-based controller is designed for optimising its performance in two independent aspects, namely collision avoidance and navigation, which are quantified as separate rewards. Instead of carefully hand-crafting a combined reward, our proposed approach trains the agent using the two rewards separately to obtain one optimal policy. It is clearly easier and more practical to design the individual rewards than manually combining them. Besides, the algorithm includes a mechanism for incorporating social norms to encourage the vehicles to follow the right-hand rule, such that they can avoid pedestrians or other vehicles in a socially acceptable manner. This is achieved by adding a continuous bias on the collision avoidance reward. Experiments using simulation environments and real robots suggest that the method is generalizable to multi-robot systems, while guaranteeing safety. In future research, we will focus on incorporating uncertainties of sensor readings for safe and reliable autonomous navigation.

Tracking a moving unmanned ground vehicle (UGV) with an autonomous Unmanned Aerial Vehicle (UAV) is challenging, particularly in GNSS-denied indoor environments where reacquiring the UGV after losing track poses a significant obstacle. This paper presents a novel learning framework designed to address these challenges, enabling a quadrotor UAV to effectively chase a moving UGV and regain tracking in an indoor environment. The proposed framework encompasses two primary components: the Track-HCL and the Tracking Vision System (TVS). The TVS leverages a lightweight tracker to offer real-time recognition and localization of the UGV. Additionally, the Chronological Ghosting (CG) method is employed to describe the UGV’s motion trend within a single frame. The Track-HCL component introduces a hybrid curriculum strategy to guide policy learning for the Deep Reinforcement Learning (DRL) agent. The Track-HCL enables the agent to learn the tracking policy conducive to target chasing and proficient reacquisition. We demonstrate the effectiveness of the proposed method in both simulation and field experiments.

This work is focused on reinforcement learning (RL)-based navigation for drones, whose localisation is based on visual odometry (VO). Such drones should avoid flying into areas with poor visual features, as this can lead to deteriorated localization or complete loss of tracking. To achieve this, we propose a hierarchical control scheme, which uses an RL-trained policy as the high-level controller to generate waypoints for the next control step and a low-level controller to guide the drone to reach subsequent waypoints. For the high-level policy training, unlike other RL-based navigation approaches, we incorporate awareness of VO performance into our policy by introducing pose estimation-related punishment. To aid robots in distinguishing between perception-friendly areas and unfavoured zones, we instead provide semantic scenes, as input for decision-making instead of raw images. This approach also helps minimise the sim-to-real application gap.

Cutter suction dredgers (CSDs) play a very important role in the construction of ports, waterways and navigational channels. Currently, most of CSDs are mainly manipulated by human operators, and a large amount of instrument data needs to be monitored in real time in case of unforeseen accidents. In order to reduce the heavy workload of the operators, we propose a data-driven offline learning approach, named Preprocessing-Prediction-Learning Control (PPLC), for obtaining the optimal control policy of the excavating operation of CSDs. The proposed framework consists of three modules, i.e., a data preprocessing module, a dynamics prediction module realized by a Convolutional Neural Network (CNN), and a deep reinforcement learning based control module. The first module is responsible for filtering out irrelevant variables through correlation analysis and dimensionality reduction of raw data. The second module works as a state transition function that provides the dynamics prediction of the excavating operation of a CSD. To realize the learning control, the third module employs the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to control the swing speed during the excavating operation. The simulation results show that the proposed framework can provide an effective and reliable solution to the automated excavating control of a CSD.

Multistep tasks, such as block stacking or parts (dis)assembly, are complex for autonomous robotic manipulation. A robotic system for such tasks would need to hierarchically combine motion control at a lower level and symbolic planning at a higher level. Recently, reinforcement learning (RL)-based methods have been shown to handle robotic motion control with better flexibility and generalizability. However, these methods have limited capability to handle such complex tasks involving planning and control with many intermediate steps over a long time horizon. First, current RL systems cannot achieve varied outcomes by planning over intermediate steps (e.g., stacking blocks in different orders). Second, the exploration efficiency of learning multistep tasks is low, especially when rewards are sparse. To address these limitations, we develop a unified hierarchical reinforcement learning framework, named Universal Option Framework (UOF), to enable the agent to learn varied outcomes in multistep tasks. To improve learning efficiency, we train both symbolic planning and kinematic control policies in parallel, aided by two proposed techniques: 1) an auto-adjusting exploration strategy (AAES) at the low level to stabilize the parallel training, and 2) abstract demonstrations at the high level to accelerate convergence. To evaluate its performance, we performed experiments on various multistep block-stacking tasks with blocks of different shapes and combinations and with different degrees of freedom for robot control. The results demonstrate that our method can accomplish multistep manipulation tasks more efficiently and stably, and with significantly less memory consumption.

We investigate the role of communication in the coordination of cooperative robot teams and its impact on performance in search and retrieval tasks. We first discuss a baseline without communication and analyse various kinds of coordination strategies for exploration and exploitation. We then discuss how the robots construct a shared mental model by communicating beliefs and/or goals with one another, as well as the coordination protocols with regard to subtask allocation and destination selection. Moreover, we also study the influence of various factors on performance including the size of robot teams, the size of the environment that needs to be explored and ordering constraints on the team goal. We use the Blocks World for Teams as an abstract testbed for simulating such tasks, where the team goal of the robots is to search and retrieve a number of target blocks in an initially unknown environment. In our experiments we have studied two main variations: a variant where all blocks to be retrieved have the same color (no ordering constraints on the team goal) and a variant where blocks of various colors need to be retrieved in a particular order (with ordering constraints). Our findings show that communication increases performance but significantly more so for the second variant and that exchanging more messages does not always yield a better team performance.