Navigating mobile robots from remote servers offers numerous benefits in industrial settings, such as centralized decision-making and powerful hardware availability. However, the QoS of outsourced controllers is heavily influenced by the communication infrastructure state. Allowing robot applications to gain insights and control these states has the potential to significantly enhance mission performance. Programmable 5G O-RAN standardized near real-time network controllers allow application domain entities to access network-specific metrics and control parameters. We propose a solution where multiple robots are connected via 5G O-RAN to a common robot application framework capable of forwarding network performance metrics and managing network resource usage. This new component allows mobile robot applications to optimize their behavior based on network conditions. The system is evaluated in a simulated environment, where necessary navigation computations can be performed either on the mobile robot's local hardware or on more powerful remote hardware. Results demonstrate enhanced mission performance in the case of network-aware navigation and further improvements when the agent's QoS is prioritized.

The ability of a robot to perceive and understand its environment is crucial for its actions and behavior. Humans are adept at using semantic information for object localization and path planning, a skill that robots need to emulate for intelligent adaptation in dynamic settings. Training of the spatial anticipation ability, which can enhance spatial perception through semantic understanding, necessitates the availability of appropriate data. Although extensive research has been conducted on datasets for outdoor environments, especially in the context of autonomous driving, there is still a notable lack of datasets specifically designed for indoor environments, with a focus on dynamic object localization. This paper introduces HabitatDyn 2.0, a dataset specifically designed for enhancing object localization capabilities with semantic information from a robot’s perspective. Besides RGB videos, semantic annotations, and depth information, HabitatDyn 2.0 also features top-down view labels for dynamic objects, which is required for training the spatial anticipation ability based on semantic information. Additionally, an algorithm that leverages spatial anticipation for dynamic object localization is presented, trained, and evaluated on the dataset.

To enable flexible quality inspection in industrial manufacturing environments, there is an increasing demand for easy-to-use automation systems that can aid factory workers in repetitive tasks. However, quality-control tasks may be too complex for a single classification algorithm to deliver results comparable to a human operator. Therefore, we propose a computer-vision architecture to improve flexibility of quality inspection tasks, that first detects objects of interest, estimates their poses and defines regions-of-interest prior to the classification task takes place. The proposed hybrid architecture consists of a deep-learning based object detection and segmentation model, and edge-based pose estimation method. We validate our approaches in texture-less, reflecting and symmetric vehicle metal sheets which are challenging for state-of-the-art object detection and pose estimation methods. Furthermore, since annotating the data required by these methods is laborious and time-consuming, we train our object detection and pose estimation architectures on 3D synthetic datasets based on available CAD model. We demonstrate promising domain generalization results on the object detection stage of our architecture (Mask R-CNN) as while trained with synthetic data, it reaches a bounding box mAP of 0.781 on synthetic and mAP of 0.686 on real images. On the other side, our simple edge-based pose estimation method can cope with texture-less parts even if using synthetic data as reference and being evaluated on real images. Our custom edge-based pose estimator reaches 16° of average rotation error and 0.14m of average translation error on real images.

In collaborative environments, real-time motion planning is crucial for industrial robots to navigate safely and efficiently. Traditional planning algorithms, such as Rapidly-exploring Random Trees (RRT) or Probabilistic Roadmaps (PRM), often face challenges in coping with dynamic environments due to their inherent computational complexity. To address this issue, we propose an approach based on Deep Reinforcement Learning (DRL) for real-time motion planning of industrial robots. Our method leverages the power of machine learning and neural networks to enable robots to make intelligent decisions in real-time, ensuring prompt and adaptive navigation. However, applying DRL to industrial robots poses unique challenges, as vision-based training is difficult and distance sensors commonly used in mobile robots are unavailable. To overcome these challenges, we employ depth cameras to generate distance information and convert the obtained point cloud into voxels using the Open3D library. The obstacles are then loaded into the simulation environment in real-time, allowing the agent to perceive and react to the dynamic environment. To achieve a low simulation-to-real-gap, we propose a hardware-in-the-loop (HIL) approach, where the real robot mimics the movements of the simulated robot. We demonstrate the effectiveness of our system through real-world experiments. Our code is available on GitHub [1].

Following up on our previous works, in this paper, we present Arena-Rosnav 2.0 an extension to our previous works Arena-Bench [1] and Arena-Rosnav [2], which adds a variety of additional modules for developing and benchmarking robotic navigation approaches. The platform is fundamentally restructured and provides unified APIs to add additional functionalities such as planning algorithms, simulators, or evaluation functionalities. We have included more realistic simulation and pedestrian behavior and provide a profound documentation to lower the entry barrier. We evaluated our system by first, conducting a user study in which we asked experienced researchers as well as new practitioners and students to test our system. The feedback was mostly positive and a high number of participants are utilizing our system for other research endeavors. Finally, we demonstrate the feasibility of our system by integrating two new simulators and a variety of state of the art navigation approaches and benchmark them against one another. The platform is openly available at https://github.com/Arena-Rosnav.

The detection of previously unseen, unexpected obstacles on the road is a major challenge for automated driving systems. Different from the detection of ordinary objects with pre-definable classes, detecting unexpected obstacles on the road cannot be resolved by upscaling the sensor technology alone (e.g., high resolution video imagers / radar antennas, denser LiDAR scan lines). This is due to the fact, that there is a wide variety in the types of unexpected obstacles that also do not share a common appearance (e.g., lost cargo as a suitcase or bicycle, tire fragments, a tree stem). Also adding object classes or adding "all" of these objects to a common "unexpected obstacle" class does not scale. In this contribution, we study the feasibility of using a deep learning video-based lane corridor (called "AI ego-corridor") to ease the challenge by inverting the problem: Instead of detecting a previously unseen object, the AI ego-corridor detects that the ego-lane ahead ends. A smart ground-truth definition enables an easy feature-based classification of an abrupt end of the ego-lane. We propose two neural network designs and research among other things the potential of training with synthetic data. We evaluate our approach on a test vehicle platform. It is shown that the approach is able to detect numerous previously unseen obstacles at a distance of up to 300 m with a detection rate of 95 %.

Over the past decades, countless autonomous navigation and dynamic obstacle avoidance approaches have been proposed by various research works. However, to bridge the gap between research and industries, these approaches are required to be extensively evaluated and benchmarked within various different setting, scenarios, and maps. However, conducting these test runs is tedious and time-consuming. Furthermore, simulation runs and test on real robots can not always cover all potentially occurring scenarios or are inaccurate in certain settings and circumstances especially when a high number of pedestrians or other dynamic entities are involved. In this paper, we propose an approach to predict the navigational performance of navigation approaches for new and unknown maps, scenarios, and robots without the necessity to conduct the actual test runs. Therefore, we acquire a large dataset consisting of thousands of evaluation runs within crowded environments from both simulation and real-world runs, which were conducted using the arena-bench platform of our previous works [1] and trained several neural network architectures to predict relevant navigational performance metrics such as collision rates or path efficiency. We demonstrate the feasibility of our neural networks by predicting the most relevant metrics with up to 95 percent accuracy compared to the groundtruth data acquired by an actual simulation run. Using this approach could prove beneficial for a number of applications and save valuable time and costs in that the performance of new navigation algorithms for crowded environments can be estimated and predicted on new maps, scenarios, and on new robots. We made the code publicly available at https://github.com/ignc-research/navprediction.

Autonomous navigation of mobile robots is an es-sential aspect in use cases such as delivery, assistance or logistics. Although traditional planning methods are well integrated into existing navigation systems, they struggle in highly dynamic en-vironments. On the other hand, Deep-Reinforcement-Learning-based methods show superior performance in dynamic obstacle avoidance but are not suitable for long-range navigation and struggle with local minima. In this paper, we propose a Deep-Reinforcement-Learning-based control switch, which has the ability to select between different planning paradigms based solely on sensor data observations. Therefore, we develop an interface to efficiently operate multiple model-based, as well as learning-based local planners and integrate a variety of state-of-the-art planners to be selected by the control switch. Subsequently, we evaluate our approach against each planner individually and found improvements in navigation performance especially for highly dynamic scenarios. Our planner was able to prefer learning-based approaches in situations with a high number of obstacles while relying on the traditional model-based planners in long corridors or empty spaces.

Assistance robots have gained widespread attention in various industries such as logistics and human assistance. The tasks of guiding or following a human in a crowded environment such as airports or train stations to carry weight or goods is still an open problem. In these use cases, the robot is not only required to intelligently interact with humans, but also to navigate safely among crowds. Thus, especially highly dynamic environments pose a grand challenge due to the volatile behavior patterns and unpredictable movements of humans. In this paper, we propose a Deep-Reinforcement-Learning-based agent for human-guiding and -following tasks in crowded environments. Therefore, we incorporate semantic information to provide the agent with high-level information like the social states of humans, safety models, and class types. We evaluate our proposed approach against a benchmark approach without semantic information and demonstrated enhanced navigational safety and robustness. Moreover, we demonstrate that the agent could learn to adapt its behavior to humans, which improves the human-robot interaction significantly.

Deploying navigation algorithms on an edge or cloud server according to the Software-as-a-Service paradigm has many advantages for autonomous mobile robots in indus-trial environments, e.g. cooperative planning and less onboard energy consumption. However, outsourcing corresponding real-time critical control functions requires a high level of reliability, which cannot be guaranteed either by modern wireless networks nor by the outsourced computing infrastructure. This work introduces redundancy concepts, which enable real-time capability within these uncertain infrastructures by providing redundant computation nodes, as well as robot-controlled switching between them. Redundancies can vary regarding their physical location, robot behavior during the switchover process and degree of activeness while quality of service concerning the primary controller is sufficient. In the case that fallback redun-dancies are not continuously active, when a disturbance occurs an initial state estimation of the robot pose has to be provided and an activation time has to be anticipated. To gain some insights on expected behavior, redundant computation nodes are deployed locally on the robot and on an outsourced computation node and consequently evaluated empirically. Quantitative and qualitative results in simulation and a real environment show that redun-dancies help to significantly improve the robot-trajectory within an unreliable network. Moreover, resource-saving redundancies, which are not continuously active, can robustly take over control by using an estimated state.

We introduce an optimization method for recent approaches on keypoint-based pose estimation of robotic manipulators utilizing monocular images. The method takes into account the segmented shape of the robot using Convolutional Neural Networks and a keypoint refinement through a set of score values. To this end, the primal 2D keypoint detection is exploited as an initial guess for further shape-based keypoint adjustments. Afterwards, the overall methods incorporates a perspective-n-point algorithm using 3D point correspondences that are derived by forward kinematics. We hereby complement an existing public dataset with annotated segmentations of a Universal Robot UR5 manipulator. The evaluation of the optimization approach shows clearly that noise on the initial key-point detection can be suppressed and minimized. Furthermore, the overall success rate of the perspective transformation can be enhanced towards more than 90%. Thus, the overall methods is applicable for single-shot pose estimation. The evaluation results also show a significant reduction of the standard deviation of the resulting pose estimation. Consequently, the proposed optimization positively affects applicability and precision.

Recently, mobile robots have become important tools in various industries, especially in logistics. Deep reinforcement learning emerged as an alternative planning method to replace overly conservative approaches and promises more efficient and flexible navigation. However, deep reinforcement learning approaches are not suitable for long-range navigation due to their proneness to local minima and lack of long term memory, which hinders its widespread integration into industrial applications of mobile robotics. In this paper, we propose a navigation system incorporating deep-reinforcement-learning- based local planners into conventional navigation stacks for long-range navigation. Therefore, a framework for training and testing the deep reinforcement learning algorithms along with classic approaches is presented. We evaluated our deep-reinforcement-learning-enhanced navigation system against various conventional planners and found that our system outperforms them in terms of safety, efficiency and robustness.

Deep Reinforcement Learning has emerged as an efficient dynamic obstacle avoidance method in highly dynamic environments. It has the potential to replace overly conservative or inefficient navigation approaches. However, integrating Deep Reinforcement Learning into existing navigation systems is still an open frontier due to the myopic nature of Deep-Reinforcement-Learning-based navigation, which hinders its widespread integration into current navigation systems. In this paper, we propose the concept of an intermediate planner to interconnect novel Deep-Reinforcement-Learning-based obstacle avoidance with conventional global planning methods using waypoint generation. Therefore, we integrate different waypoint generators into existing navigation systems and compare the joint system against traditional ones. We found an increased performance in terms of safety, efficiency and path smoothness, especially in highly dynamic environments.

The imagination of the surrounding environment based on the experience and semantic cognition has great potential to extend the limited observations to leverage the ability for mapping, collision avoidance and path planning. This paper provides a training-based algorithm for mobile robots to perform spatial imagination based on semantic cognition and evaluates the proposed method for the mapping task. We utilize a photo-realistic simulation environment, Habitat, for training and evaluation. The trained model is composed of Resent-18 as encoder and U-net as the backbone. We demonstrate that the algorithm can perform imagination for unseen parts of the object universally, by recalling the images and experience and compare our approach with traditional semantic mapping methods. It is found that our approach will improve the efficiency and accuracy of semantic mapping.

Autonomous mobile transport systems are a key enabler for flexible production organization. In order to enhance the software life cycle management and the overall function range, service-based offloading of software function towards cloud and edge is a valid alternative to monolithic onboard software architectures. We introduce an approach towards an edge-computing-based improvement of the classic autonomous navigation stack in terms of considering semantic policies using visual object detection and localization. Thus, context-aware navigation in regards to safety and efficiency can be implemented, e.g. to fulfill directions of standards and guidelines for production, hospital or public domains. Our semantic path planning is implemented as an addition or as a replacement of common local planning services following a microservice approach. The semantic optimization of the initial trajectory is successfully shown in regards to the following policies: keeping to the right side of the surrounding environment, avoiding to drive near to closed doors and passing humans on the right side. In addition, we reveal implementation of further policies by adapting the optimization policies and show within a scenario-based evaluation that the usage of edge computing in comparison to onboard computing yields performance gains.

Augmented Reality and mobile robots are gaining increased attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this account, we propose a novel approach to calibrate Augmented Reality devices using 3D depth sensor data. We use the depth camera of a Head Mounted Augmented Reality Device, the Microsoft Hololens, for deep learning-based calibration. Therefore, we modified a neural network based on the recently published VoteNet architecture which works directly on raw point cloud input observed by the Hololens. We achieve satisfying results and eliminate external tools like markers, thus enabling a more intuitive and flexible work flow for Augmented Reality integration. The results are adaptable to work with all depth cameras and are promising for further research. Furthermore, we introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.

Autonomous mobile transport systems are a key enabler for flexible production organization. In order to enhance the software life cycle management and the overall function range, service-based offloading of software function towards cloud and edge is a valid alternative to monolithic onboard software architectures. We introduce an approach towards an edge-computing-based improvement of the classic autonomous navigation stack in terms of considering semantic policies using visual object detection and localization. Thus, context-aware navigation in regards to safety and efficiency can be implemented, e.g. to fulfill directions of standards and guidelines for production, hospital or public domains.

Mobile autonomous transport systems are a crucial part of flexible factory logistics enabling an adaptable industrial production. Whereas connectivity of these mobile robots is still mostly covered through Wifi, applicators suffer from interferences and unreliableness due to the harsh environmental conditions. We present a case study using 4G campus networks applying network slicing technology and edge computing in order to realize a distributed control scenario. Control algorithms are offloaded to the edge following a navigation as a service approach. Basic safety functions remain onboard while self-localization and mapping as well as motion planning run on either a factory edge, a nearby edge or on a public cloud. We present an evaluation of the overall architecture and focus on effects of offloading control functions towards the edge. Finally, we provide an outlook towards the usage of 5G.

Human robot collaboration is aspiring to establish hybrid work environments in accordance with specific strengths of humans and robots. We present an approach of flexibly integrating robotic handover assistance into collaborative assembly tasks through the use of natural communication. For flexibly instructed handovers, we implement recent Convolutional Neural Networks in terms of object detection and grasping of arbitrary objects based on an RGB-D camera equipped to a robot following the eye-in-hand principle. In order to increase fluency and efficiency of the overall assembly process, we investigate the human ability to instruct the robot predictively with voice commands. We conduct a user study quantitatively and qualitatively evaluating the predictive instruction in order to achieve just-in-time handovers of tools needed for following subtasks. We compare our predictive strategy with a pure manual assembly having all tools in direct reach and a step-by-step reactive handover. The results reveal that the human is able to predict the handover comparable to algorithm-based predictors. Nevertheless, human prediction does not rely on extensive prior knowledge and is thus suitable for more flexible usage. However, the cognitive workload for the worker is increased compared to manual or reactive assembly.

The recently applied General Data Protection Regulation (GDPR) aims to protect all EU citizens from privacy and data breaches in an increasingly data-driven world. Consequently, this deeply affects the factory domain and its human-centric automation paradigm. Especially collaboration of human and machines as well as individual support are enabled and enhanced by processing audio and video data, e.g. by using algorithms which re-identify humans or analyse human behaviour. We introduce most significant impacts of the recent legal regulation change towards the automations domain at a glance. Furthermore, we introduce a representative scenario from production, deduce its legal affections from GDPR resulting in a privacy-aware software architecture. This architecture covers modern virtualization techniques along with authorization and end-to-end encryption to ensure a secure communication between distributes services and databases for distinct purposes.

We present a hybrid programming method for industrial robots combining advantages of manual haptic guidance of the end-effector and programming approaches using non-haptic pointing gestures for the spatial definition of poses and trajectories. Whereas the bare-hand spatial interaction can be implemented and performed cost- and time-efficiently but lacks accuracy, haptic-interaction is more time-consuming but it is used in a reduced manner in order to enable a highly-accurate refinement of target working poses. Additionally, the user is supported by a mobile Augmented Reality simulation providing spatial validation of the robot program, program management and transmission towards the robot controller. The implementation is realized by a compliance control based on a sensor mounted between flange and end-effector combined with our former introduced approach for spatial programming. We conducted a user study comparing Teach-In and Offline programming. The analysis shows a significant reduction of programming duration as well as a reduction of programming errors compared with Teach-In. Most participants favor the hybrid programming system. No significant differences for the programming duration could be determined between experts and non-experts. In comparison between haptic and non-haptic interaction, non-experts favor non-haptic interaction due to the higher intuitiveness of pointing gestures compared to direct physical interaction.

We propose an approach for integrated object and path demonstration. The main idea is to choose the motion of the robot depending on the object it grabs. When programming, an object is fixed at the robot's gripper. Path programming is done using haptic guidance. Afterwards, if a robot is handed over an object it knows before, it reproduces the path assigned to the object. Path programming and object recognition can be done with a single force/torque sensor. In our opinion, this way of programming is especially suited for creating pick and place tasks in frequently changing production environments where (re-)programming has to be done frequently.

We have developed a spatial programming system for industrial robots based on gestures and Augmented Reality. In this respect, we aim for a markerless and robust finger gesture recognition on the handheld devices. The user should be enabled to interact with virtual objects representing the robot program in front of the device. The great challenge is to be fast and robust in terms of diffuse background and changing environmental conditions. In this contribution, we present an image-based finger gesture recognition approach which is tailored to efficient implementation on mobile devices, e.g. Smartphones and Tablet PCs. For this purpose, we introduce an adaptive, rule-based skin classifier. The tracking is done by a specific particle filter: the Condensation algorithm. Due to further implemented improvements, we reduce the number of particles in order to reduce the computational time of the algorithm. The gesture recognition is implemented in an Android app, enabling the user to translate and rotate virtual object within an Augmented Reality application in real-time.

The presented spatial programming system provides an assistance system for online programming of industrial robots. A handheld device and a motion tracking system establish the basis for a modular 3D programming approach corresponding to different phases of robot programming: definition, evaluation and adaption. Static and dynamic gestures enable the program definition of poses, trajectories and tasks. The spatial evaluation is done using an Augmented Reality application on a handheld device. Therefore, the programmer is able to move freely within the robot cell and define the program spatially through gestures. The camera image of the handheld is simultaneously enhanced by virtual objects representing the robot program. Based on 3D motion tracking of human movements and a mobile Augmented Reality application, we introduce a novel kind of interaction for the adaption of robot programs. The programmer is enabled to interact with virtual program components through bare-hand gestures. Such sample forms of interaction include translation and rotation applicable to poses, trajectories or tasks representations. Finally, the program is adapted according to the gestural changes and can be transferred from the handheld device directly to the robot controler.

Gesture-based control systems for industrial robotics benefit from a natural form of human machine interaction. Among the high costs of such systems, there are high requirements for safety, robustness of the gesture recognition and general applicability in the industrial environment. Novel consumer electronic sensors provide a robust recognition, reliability and adequate accuracies at low costs. We present our work on gesture-based control and programming with an emphasis on a natural definition of complex trajectories. Besides standard online programming methods, we introduce a programming technique, which enables the worker to define a movement by his body parts, e.g. the hand. Imitating the human movement, the robot can move its end-effector relatively from a starting point.