Building upon the foundations laid by our previous work, this paper introduces Arena 4.0, a significant advancement of Arena 3.0 [1], Arena-Bench [2], Arena 1.0 [3], and Arena 2.0 [4]. Arena 4.0 provides three main novel contributions: 1) a generative-model-based world and scenario generation approach using large language models (LLMs) and diffusion models, to dynamically generate complex, humancentric environments from text prompts or 2D floorplans that can be used for development and benchmarking of social navigation strategies. 2) A comprehensive 3D model database which can be extended with 3D assets and semantically linked and annotated using a variety of metrics for dynamic spawning and arrangements inside 3D worlds. 3) The complete migration towards ROS 2, which ensures operation with state-of-the-art hardware and functionalities for improved navigation, usability, and simplified transfer towards real robots. We evaluated the platforms performance through a comprehensive user study and its world generation capabilities for benchmarking demonstrating significant improvements in usability and efficiency compared to previous versions. Arena 4.0 is openly available at https://github.com/Arena-Rosnav.

Social navigation has become increasingly important for robots operating in human environments, yet many newly proposed navigation methods remain narrowly tailored or exist only as proof-of-concept prototypes. Building on our previous work with Arena, a social navigation development platform, we now propose, Arena-Bench 2.0 a comprehensive social navigation benchmark of state-of-the-art planners, fully integrated into the Arena framework. To achieve this, we developed a novel plugin structure—implemented on ROS2—to streamline the integration process and ensure straightforward, efficient workflows. As a demonstration, we integrated various learning-based and model-based navigation approaches and constructed a diverse set of social navigation scenarios to rigorously evaluate each planner. Specifically, we introduce a scenario generation node that allows users to construct complex, realistic social contexts through a web-based interface. We subsequently perform an extensive benchmark of all integrated planners, assessing both navigational and social metrics. Our evaluation also considers factors such as sensor input, reaction time, and latency, enabling insights into which planner may be most appropriate under different circumstances. The findings offer valuable guidance for selecting suitable planners for specific scenarios. The code is publicly available at https://github.com/Arena-Rosnav.

The ability of a robot to perceive and understand its environment is crucial for its actions and behavior. Humans are adept at using semantic information for object localization and path planning, a skill that robots need to emulate for intelligent adaptation in dynamic settings. Training of the spatial anticipation ability, which can enhance spatial perception through semantic understanding, necessitates the availability of appropriate data. Although extensive research has been conducted on datasets for outdoor environments, especially in the context of autonomous driving, there is still a notable lack of datasets specifically designed for indoor environments, with a focus on dynamic object localization. This paper introduces HabitatDyn 2.0, a dataset specifically designed for enhancing object localization capabilities with semantic information from a robot’s perspective. Besides RGB videos, semantic annotations, and depth information, HabitatDyn 2.0 also features top-down view labels for dynamic objects, which is required for training the spatial anticipation ability based on semantic information. Additionally, an algorithm that leverages spatial anticipation for dynamic object localization is presented, trained, and evaluated on the dataset.

In collaborative environments, real-time motion planning is crucial for industrial robots to navigate safely and efficiently. Traditional planning algorithms, such as Rapidly-exploring Random Trees (RRT) or Probabilistic Roadmaps (PRM), often face challenges in coping with dynamic environments due to their inherent computational complexity. To address this issue, we propose an approach based on Deep Reinforcement Learning (DRL) for real-time motion planning of industrial robots. Our method leverages the power of machine learning and neural networks to enable robots to make intelligent decisions in real-time, ensuring prompt and adaptive navigation. However, applying DRL to industrial robots poses unique challenges, as vision-based training is difficult and distance sensors commonly used in mobile robots are unavailable. To overcome these challenges, we employ depth cameras to generate distance information and convert the obtained point cloud into voxels using the Open3D library. The obstacles are then loaded into the simulation environment in real-time, allowing the agent to perceive and react to the dynamic environment. To achieve a low simulation-to-real-gap, we propose a hardware-in-the-loop (HIL) approach, where the real robot mimics the movements of the simulated robot. We demonstrate the effectiveness of our system through real-world experiments. Our code is available on GitHub [1].

Following up on our previous works, in this paper, we present Arena-Rosnav 2.0 an extension to our previous works Arena-Bench [1] and Arena-Rosnav [2], which adds a variety of additional modules for developing and benchmarking robotic navigation approaches. The platform is fundamentally restructured and provides unified APIs to add additional functionalities such as planning algorithms, simulators, or evaluation functionalities. We have included more realistic simulation and pedestrian behavior and provide a profound documentation to lower the entry barrier. We evaluated our system by first, conducting a user study in which we asked experienced researchers as well as new practitioners and students to test our system. The feedback was mostly positive and a high number of participants are utilizing our system for other research endeavors. Finally, we demonstrate the feasibility of our system by integrating two new simulators and a variety of state of the art navigation approaches and benchmark them against one another. The platform is openly available at https://github.com/Arena-Rosnav.

The detection of previously unseen, unexpected obstacles on the road is a major challenge for automated driving systems. Different from the detection of ordinary objects with pre-definable classes, detecting unexpected obstacles on the road cannot be resolved by upscaling the sensor technology alone (e.g., high resolution video imagers / radar antennas, denser LiDAR scan lines). This is due to the fact, that there is a wide variety in the types of unexpected obstacles that also do not share a common appearance (e.g., lost cargo as a suitcase or bicycle, tire fragments, a tree stem). Also adding object classes or adding "all" of these objects to a common "unexpected obstacle" class does not scale. In this contribution, we study the feasibility of using a deep learning video-based lane corridor (called "AI ego-corridor") to ease the challenge by inverting the problem: Instead of detecting a previously unseen object, the AI ego-corridor detects that the ego-lane ahead ends. A smart ground-truth definition enables an easy feature-based classification of an abrupt end of the ego-lane. We propose two neural network designs and research among other things the potential of training with synthetic data. We evaluate our approach on a test vehicle platform. It is shown that the approach is able to detect numerous previously unseen obstacles at a distance of up to 300 m with a detection rate of 95 %.

Over the past decades, countless autonomous navigation and dynamic obstacle avoidance approaches have been proposed by various research works. However, to bridge the gap between research and industries, these approaches are required to be extensively evaluated and benchmarked within various different setting, scenarios, and maps. However, conducting these test runs is tedious and time-consuming. Furthermore, simulation runs and test on real robots can not always cover all potentially occurring scenarios or are inaccurate in certain settings and circumstances especially when a high number of pedestrians or other dynamic entities are involved. In this paper, we propose an approach to predict the navigational performance of navigation approaches for new and unknown maps, scenarios, and robots without the necessity to conduct the actual test runs. Therefore, we acquire a large dataset consisting of thousands of evaluation runs within crowded environments from both simulation and real-world runs, which were conducted using the arena-bench platform of our previous works [1] and trained several neural network architectures to predict relevant navigational performance metrics such as collision rates or path efficiency. We demonstrate the feasibility of our neural networks by predicting the most relevant metrics with up to 95 percent accuracy compared to the groundtruth data acquired by an actual simulation run. Using this approach could prove beneficial for a number of applications and save valuable time and costs in that the performance of new navigation algorithms for crowded environments can be estimated and predicted on new maps, scenarios, and on new robots. We made the code publicly available at https://github.com/ignc-research/navprediction.

Autonomous navigation of mobile robots is an es-sential aspect in use cases such as delivery, assistance or logistics. Although traditional planning methods are well integrated into existing navigation systems, they struggle in highly dynamic en-vironments. On the other hand, Deep-Reinforcement-Learning-based methods show superior performance in dynamic obstacle avoidance but are not suitable for long-range navigation and struggle with local minima. In this paper, we propose a Deep-Reinforcement-Learning-based control switch, which has the ability to select between different planning paradigms based solely on sensor data observations. Therefore, we develop an interface to efficiently operate multiple model-based, as well as learning-based local planners and integrate a variety of state-of-the-art planners to be selected by the control switch. Subsequently, we evaluate our approach against each planner individually and found improvements in navigation performance especially for highly dynamic scenarios. Our planner was able to prefer learning-based approaches in situations with a high number of obstacles while relying on the traditional model-based planners in long corridors or empty spaces.

Assistance robots have gained widespread attention in various industries such as logistics and human assistance. The tasks of guiding or following a human in a crowded environment such as airports or train stations to carry weight or goods is still an open problem. In these use cases, the robot is not only required to intelligently interact with humans, but also to navigate safely among crowds. Thus, especially highly dynamic environments pose a grand challenge due to the volatile behavior patterns and unpredictable movements of humans. In this paper, we propose a Deep-Reinforcement-Learning-based agent for human-guiding and -following tasks in crowded environments. Therefore, we incorporate semantic information to provide the agent with high-level information like the social states of humans, safety models, and class types. We evaluate our proposed approach against a benchmark approach without semantic information and demonstrated enhanced navigational safety and robustness. Moreover, we demonstrate that the agent could learn to adapt its behavior to humans, which improves the human-robot interaction significantly.

Recently, mobile robots have become important tools in various industries, especially in logistics. Deep reinforcement learning emerged as an alternative planning method to replace overly conservative approaches and promises more efficient and flexible navigation. However, deep reinforcement learning approaches are not suitable for long-range navigation due to their proneness to local minima and lack of long term memory, which hinders its widespread integration into industrial applications of mobile robotics. In this paper, we propose a navigation system incorporating deep-reinforcement-learning- based local planners into conventional navigation stacks for long-range navigation. Therefore, a framework for training and testing the deep reinforcement learning algorithms along with classic approaches is presented. We evaluated our deep-reinforcement-learning-enhanced navigation system against various conventional planners and found that our system outperforms them in terms of safety, efficiency and robustness.

Deep Reinforcement Learning has emerged as an efficient dynamic obstacle avoidance method in highly dynamic environments. It has the potential to replace overly conservative or inefficient navigation approaches. However, integrating Deep Reinforcement Learning into existing navigation systems is still an open frontier due to the myopic nature of Deep-Reinforcement-Learning-based navigation, which hinders its widespread integration into current navigation systems. In this paper, we propose the concept of an intermediate planner to interconnect novel Deep-Reinforcement-Learning-based obstacle avoidance with conventional global planning methods using waypoint generation. Therefore, we integrate different waypoint generators into existing navigation systems and compare the joint system against traditional ones. We found an increased performance in terms of safety, efficiency and path smoothness, especially in highly dynamic environments.

The imagination of the surrounding environment based on the experience and semantic cognition has great potential to extend the limited observations to leverage the ability for mapping, collision avoidance and path planning. This paper provides a training-based algorithm for mobile robots to perform spatial imagination based on semantic cognition and evaluates the proposed method for the mapping task. We utilize a photo-realistic simulation environment, Habitat, for training and evaluation. The trained model is composed of Resent-18 as encoder and U-net as the backbone. We demonstrate that the algorithm can perform imagination for unseen parts of the object universally, by recalling the images and experience and compare our approach with traditional semantic mapping methods. It is found that our approach will improve the efficiency and accuracy of semantic mapping.

Augmented Reality and mobile robots are gaining increased attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this account, we propose a novel approach to calibrate Augmented Reality devices using 3D depth sensor data. We use the depth camera of a Head Mounted Augmented Reality Device, the Microsoft Hololens, for deep learning-based calibration. Therefore, we modified a neural network based on the recently published VoteNet architecture which works directly on raw point cloud input observed by the Hololens. We achieve satisfying results and eliminate external tools like markers, thus enabling a more intuitive and flexible work flow for Augmented Reality integration. The results are adaptable to work with all depth cameras and are promising for further research. Furthermore, we introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.