Mobile robots can perform increasingly impressive feats in controlled environments. Many real applications, though, especially for walking robots, introduce a high degree of unforeseen difficulties, yet require very robust robot operation. In these cases, it is still often not possible to guarantee the needed reliability.We present an approach to utilize unsupervised anomaly detection to implement a fear-based adaptation of robot behavior. This allows robots to automatically and quickly react to any type of unexpected problems. Neither the environment nor the type of disturbance has to be known beforehand, as the system requires only a small amount of baseline data for training, which can be collected in a laboratory environment. Additionally, it can work on arbitrary robot hardware and be integrated in all types of robot control structures.We evaluated our approach in simulation and on state of the art walking robots, ANYmal, Spot and our own six-legged walking robot prototype, in a realistic field test environment in the Tabernas desert in Spain. Our results showcase that we can quickly detect arbitrary problems based on significantly different types of sensor data and decrease robot fall rates in the most extreme scenarios from 56% to 4%. This promises significant increases in robustness for all types of walking robots in highly challenging and previously unknown environments.

In real-world few-shot image classification tasks the lack of abundant data makes training and testing very challenging. The classification model must learn the most meaningful features using only a few sample images without context knowledge. Here, interpretability methods for deep models are helpful for increased comprehensibility and verification. However, these advantages are limited without the ability to correct the model directly. Therefore, we propose an interpretable approach for few-shot object recognition that includes optional interactive teaching to close the feedback loop. We leverage pretrained vision transformers as backbones and a part-based inference particularly favors interpretability. We use a visual concept bank to translate semantic visual features between the human and the model. Even without any human interaction, our model performs competitively compared to state-of-the-art methods in few-shot image classification tasks. Beyond that, we demonstrate the benefits of our interactive interfaces. We show how they can significantly improve the robustness in fine-grained recognition tasks and help to quickly adapt the model without complex fine-tuning.

While individual robots are becoming increasingly capable, the complexity of expected missions increases exponentially in comparison. To cope with this complexity, heterogeneous teams of robots have become a significant research interest in recent years. Making effective use of the robots and their unique skills in a team is challenging. Dynamic runtime conditions often make static task allocations infeasible, requiring a dynamic, capability-aware allocation of tasks to team members. To this end, we propose and implement a system that allows a user to specify missions using Behavior Trees (BTs), which can then, at runtime, be dynamically allocated to the current robot team. The system allows to statically model an individual robot’s capabilities within our ros_bt_py BT framework. It offers a runtime auction system to dynamically allocate tasks to the most capable robot in the current team. The system leverages utility values and pre-conditions to ensure that the allocation improves the overall mission execution quality while preventing faulty assignments. To evaluate the system, we simulated a find-and-decontaminate mission with a team of three heterogeneous robots and analyzed the utilization and overall mission times as metrics. Our results show that our system can improve the overall effectiveness of a team while allowing for intuitive mission specification and flexibility in the team composition.

Deciding where to go next is a challenging task for humans. However, for robots in unknown environments, this becomes even more demanding. In planetary explorations, the robots are continuously challenged with the task of exploring novel areas, yet so far, humans decide for the robots where to go. Even then, prioritizing the next target based on previous knowledge is complex. In our proposed work, the robot utilizes data about its surroundings from drone or satellite images. Alternatively, a volumetric representation can be reduced to form a suitable input. From the input, tiles are selected and embedded by different autoencoder variants. The robot can select the most promising next exploration goal through the distance in the embedding to the previous samples. In this work, a variational autoencoder, a Wasserstein autoencoder, and a spherical autoencoder are evaluated against each other. The latter two variants yield a high information gain when evaluated on satellite data from the Netherlands. Additionally, the framework was employed on data from an analog mission in the Tabernas desert. Through the framework, the robots get an understanding of which goals yield the most information gain and, therefore, can quickly improve their knowledge about their surroundings.

As intelligent systems become increasingly important in our daily lives, new ways of interaction are needed. Classical user interfaces pose issues for the physically impaired and are partially not practical or convenient. Gesture recognition is an alternative, but often not reactive enough when conventional cameras are used. This work proposes a Spiking Convolutional Neural Network, processing event- and depth data for gesture recognition. The network is simulated using the open-source neuromorphic computing framework LAVA for offline training and evaluation on an embedded system. For the evaluation three open source data sets are used. Since these do not represent the applied bi-modality, a new data set with synchronized event- and depth data was recorded. The results show the viability of temporal encoding on depth information and modality fusion, even on differently encoded data, to be beneficial to network performance and generalization capabilities.

The use of mobile robots has become increasingly common in multiple areas of daily life. To increase their autonomy for performing various tasks, efficient navigation skills are essential. The most crucial component of such navigation is the ability to calculate a global path between two points. The global path planning problem for mobile robots is typically limited to two-dimensional environments, in which the environment is projected onto a planar surface. While this approach works well in structured environments like industrial settings, it may not be suitable for all applications of mobile robots. With modern walking robots, capable of navigating complex terrain, more advanced path planning approaches are necessary. This work proposes a path-planning approach that utilizes the entire three-dimensional space, allowing for navigation in even the most challenging terrain. The central idea is to extend a traditional A* path planner to work directly on a fast volumetric map structure to generate optimal paths through the environment. Multiple optimizations and adjustments are introduced to improve the algorithm’s performance. By applying morphology operators to sparse maps, sensor inaccuracies during the map construction are mitigated. Additionally, adjustments are made to handle the added complexity introduced by the extra search space dimension and to comply with the limitations of autonomous walking robots. This is paired with an efficient caching strategy to enhance the overall path-planning speed. The capability of the path planning approach is evaluated using both artificial and real-world maps. The results demonstrate that this approach shows great potential for enabling mobile ground robots to autonomously navigate even the most demanding terrains utilizing the entire three-dimensional space.

Autonomous Mobile Robots are used with increasing frequency in inspection and maintenance tasks completing fixed goal sequences. The downtime robots experience between goals offers an opportunity to gather additional environment information instead of resting. Uncertainty in the amount of downtime available rules out the definition of a pre-determined schedule set by an external operator. Instead, the robot itself should decide dynamically, what information it should gather before its next task begins. This results in a multi-objective optimization problem trying to maximize information gain while utilizing as much of the available time as possible. We propose a genetic algorithm to solve the presented optimization problem and introduce two different models for artificial curiosity used inside the fitness function for gathering as much information as possible. For planning the genetic algorithm utilizes a multi-map approach using information and obstacle maps. We evaluated our models in a pre-defined and pre-mapped Gazebo environment with a given information map and evaluated their performance against an information-agnostic coverage algorithm. In this work, we show that utilizing artificial curiosity in path planning can result in major information gains by effectively using downtime.

Motion planning for mobile robot platforms is one of the long-established research fields in robotics. In this paper, we propose a trajectory planner for mobile holonomic robots to steer non-holonomic conventional passive wheelchairs in dynamic environments. The challenges to overcome when steering a wheelchair are to find smooth feasible trajectories, maintain a fast reactive response to dynamic obstacles and to satisfy a set of additional constraints such as limiting physical forces acting on the wheelchair occupants. Our approach is a variant of the timed-elastic-bands (TEB) planner, which includes a footprint of the wheelchair during optimization, and generates a steering angle which is then consumed by an arm controller to actuate the relative orientation between the wheelchair and the mobile platform. This is realized by posing new non-holonomic and kinodynamic constraints on the TEB planner and an implementation of a suitable real-time dual-arm controller for executing steering commands. We demonstrate our results based on a TEB baseline comparison in simulation using functional models of our robot HoLLiE and a wheelchair.

Exploring unknown environments, such as caves or planetary surfaces, requires a quick understanding of the surroundings. Beforehand, only aerial footage from satellites or images from previous missions might be available. The proposed ensemble based anomaly detection framework utilizes previously gained knowledge and incorporates it with insights gained during the mission. The modular system consists of different networks which are combined to determine anomalies in the current surroundings. By utilizing data from other missions, simulations or aerial photos, a precise anomaly detection can be achieved at the start of a mission. The system can further be improved by training new networks during the mission, which can be incorporated into the ensemble at runtime. This allows for synchronous execution of mission and training of models on a base station. The proposed system is tested and evaluated on an ANYmal C walking robot in different scenarios, however the approach is applicable for different kinds of mobile robots. The results show a clear improvement of ensembles compared to individual networks, while keeping a small memory footprint and low inference time on the mobile system.

Complex robots in challenging scenarios require constant monitoring of their state and adaptation of their behavior to ensure robustness, reliability and longevity. While known possible errors can be specifically surveilled, other prob-lems can be fully unforeseen, requiring detection systems able to identify novel faults. We detect possible faults as anomalies on various internal sensor data, utilizing unsupervised learning techniques. A bidirectional Wasserstein GAN approach for anomaly detection on multivariate, highly dependent time-series data is implemented and trained on a small amount of non-anomalous robot sensor data. This model is then used for inference on the on-board hardware of a robot without parallel processing units. We evaluate multiple variants of the architecture using manually introduced anomalies in the form of different weights attached to the robot's legs. Overall we are able to show that RoBiGAN is able to consistently detect and localize small anomalies in an online scenario, with little to no robot specific modeling needed.

We present a biologically inspired approach for path planning with dynamic obstacle avoidance. Path plan-ning is performed in a condensed configuration space of a robot generated by self-organizing neural networks (SONN). The robot itself and static as well as dynamic obstacles are mapped from the Cartesian task to the configuration space by precomputed kinematics. The condensed space represents a cognitive map of the environment, which is inspired by place cells and the concept of cognitive maps in mammalian brains. Generation of training data as well as the evaluation are performed on a real industrial robot accompanied by simulations. To evaluate reactive collision-free online planning within a changing environment, a demonstrator was realized. Then, a comparative study regarding sample-based planners was carried out. The robot is able to operate in dynamically changing environments and re-plan its motion trajectories within impressing 0.02 seconds, which proofs the real-time capability of our concept.

Speech is a convenient hands-free communication channel where humans are already experienced users. It can implicitly create trustfulness between two operators and lead to a comfortable and natural collaborative environment. As stated in existing literature, speech interaction could increase efficiency and improve certain aspects of Human-Robot Collaboration (HRC). Anyway, speech recognition in industrial scenarios presents different challenges: the typical noisy environment can affect dramatically the interaction performance, leading to an unacceptable inaccuracy in understanding the uttered intention.In this work, we propose and evaluate a modular system for robust and natural speech interaction in challenging acoustical environments. The system has been integrated and tested in a realistic HRC scenario in which the acoustic interaction and efficiency have been evaluated. The developed framework focuses on decreasing the requirements in terms of signal-to-noise ratio, providing a methodology to evaluate the naturalness of the interaction and improvements in efficiency.The solution is designed with a modular approach, providing an easy configuration for ROS-based systems. In this way, it allows a simple integration in existing applications and future research projects, where a dual speech-based interaction can increase the overall performance of the HRC.

Field and service robots that do not understand the hazards in their environment limit their potential by acting overly careful or navigating into potentially dangerous areas. We present an extended modular mapping framework which allows to model different types of hazards from a multitude of inputs. The proposed approach is generalized for storage of arbitrary data, therefore not limiting the usage to one use case. Furthermore the framework allows the fusion of risks to calculate the overall risk for an individual robot from its surroundings. The system was tested with LAURON V, a hexapod designed for walking over rough and hazardous terrain.

Robotic automation is a key driver for the advancement of technology. The skills of human workers, however, are difficult to program and seem currently unmatched by technical systems. In this work we present a data-driven approach to extract and learn robot-independent contact skills from human demonstrations in simulation environments, using a Long Short Term Memory (LSTM) network. Our model learns to generate error-correcting sequences of forces and torques in task space from object-relative motion, which industrial robots carry out through a Cartesian force control scheme on the real setup. This scheme uses forward dynamics computation of a virtually conditioned twin of the manipulator to solve the inverse kinematics problem. We evaluate our methods with an assembly experiment, in which our algorithm handles part tilting and jamming in order to succeed. The results show that the skill is robust towards localization uncertainty in task space and across different joint configurations of the robot. With our approach, non-experts can easily program force-sensitive assembly tasks in a robot-independent way.

Bio-inspired robots take advantage of millions of years of evolution to provide interesting and flexible solutions for issues related to motion and perception. Often, they have challenging kinematics classical robotics control mechanisms are not always able to take advantage of them. A concrete example of this is LAURON V, a six-legged robot for space exploration inspired by the stick insects. The main goals of this work is to combine classical behaviour-based control with motor primitives implemented with SNN for motion representation. We extend a previously presented bio-inspired approach to represent hand and arm motion using motor primitives, and combine it with a behaviour-based architecture to model different locomotion behaviours for a multi-legged robot. There are four main components. First, to model the individual leg motions we use two motor primitives implemented with spiking neural networks for the swing and stance phases. Second, to control the motor primitives of each leg there are local behaviours corresponding to each phase, and corresponding to each activation pattern. Third, the activation patterns are used to facilitate multi-leg coordination and generate different walking behaviours. Fourth, a high-level control interface integrates control signals from other sources and activates the patterns. We conducted five different experiments to evaluate our approach in a simulated environment using the Neurorobotics Platform (NRP). The results show that our modelling approach with motor primitives is flexible enough to represent different types of motions, and also highlight the value of the NRP for robotics development.

Most robots consistently repeat their motion with- out changes in a precise and consistent manner. But nowadays there are also robots able to dynamically change their motion and plan according to the people and environment that surround them. Furthermore, they are able to interact with humans and cooperate with them. With no information about the robot targets and intentions, the user feels uncomfortable even with a safe robot. In close human-robot collaboration, it is very important to make the user able to understand the robot intentions in a quick and intuitive way. In this work we have developed a system to use augmented reality to project directly into the workspace useful information. The robot intuitively shows its planned motion and task state. The AR module interacts with a vision system in order to display the changes in the workspace in a dynamic way. The representation of information about possible collisions and changes of plan allows the human to have a more comfortable and efficient interaction with the robot. The system is evaluated in different setups.

Nowadays robots are able to work safely close to humans. They are light-weight, intrinsically safe and capable of avoiding obstacles as well as understand and predict human motions. In this collaborative scenario, the communication between humans and robots is a fundamental aspect to achieve good efficiency and ergonomics in the task execution. A lot of research has been made related to robot understanding and prediction of the human behavior, allowing the robot to replan its motion trajectories. This work is focused on the communication of the robot's intentions to the human to make its goals and planned trajectories easily understandable. Visual and acoustic information has been added to give the human an intuitive feedback to immediately understand the robot's plan. This allows a better interaction and makes the humans feel more comfortable, without any feeling of anxiety related to the unpredictability of the robot motion. Experiments have been conducted in a collaborative assembly scenario. The results of these tests were collected in questionnaires, in which the humans reported the differences and improvements they experienced using the feedback communication system.

Compliant end effectors in robotics are an important prerequisite for the field of object manipulation and environment interactions. However, current manipulators are usually stiff position-controlled systems, generating the need to add compliance. In this work, we present Forward Dynamics Compliance Control (FDCC), a new threefold control concept that realizes Cartesian compliance through combining Admittance, Impedance and Force Control into one control strategy. We close the control loop only through a force-torque sensor, allowing a system independent and decoupled configuration of the end effector compliance. As a key component in FDCC, we leverage forward dynamics simulations of a virtual model to directly map Cartesian inputs to joint control commands, leading to excellent stability in singularities. Experiments on three different robotic manipulators verify the key advantages of this approach.

In this work, the ReApp Engineering Workbench and its underlying semantically enriched app models are presented. The usage of a model, which describes the apps functionality, interfaces and other attributes, allows the utilization of engineering tools for code generation and automated testing. Further, it ensures the compatibility of the generated interfaces, which in turn enhances the reusability of the developed apps in larger applications.

New sensor systems, efficient planning algorithms and increased computational power have led to a growing interest in high-resolution 3-D data for challenging applications like mobile manipulation, 3-D mapping and object recognition. 3-D laser scanners built using a rotating or swivelling 2-D laser scanner are a widespread technology for obtaining 3-D point clouds, but convenient calibration methods are needed to increase the sensor precision and loosen the requirements on mechanical tolerances. We present an approach for the automatic self-calibration of such scanners, using only a single scan of a targetless environment recorded over a complete 360° rotation of the external motor that controls the motion of the 2-D sensor. By exploiting the intrinsic redundancies of the recorded point clouds and decimating the clouds using a voxel grid filter, we are able to compute and optimize a point cloud quality measure very quickly and without relying on explicit calibration targets. Our results on simulated and real data show the effectiveness of our approach by creating high-quality 3-D point clouds.

This paper gives an overview on our framework for efficient collision detection in robotic applications. It unifies different data structures and algorithms that are optimized for Graphics Processing Unit (GPU) architectures. A speed-up in various planning scenarios is achieved by utilizing storage structures that meet specific demands of typical use-cases like mobile platform planning or full body planning. The system is also able to monitor the execution of motion trajectories for intruding dynamic obstacles and triggers a replanning or stops the execution. The presented collision detection is deployed in local dynamic planning with live pointcloud data as well as in global a-priori planning. Three different mobile manipulation scenarios are used to evaluate the performance of our approach.

Multi-legged walking robots often make use of sophisticated control architectures to play their strengths in rough and unknown environments. The adaptability of these robots is an essential skill to achieve the maneuverability and autonomy needed in their application fields. In this work we present a reactive control approach for the hexapod LAURONV, which enables it to overcome large obstacles and steep slopes without any knowledge about the environment. A key to this success can also be seen in the increased kinematic adaptability due to the fourth rotational joint in the bio-inspired leg kinematics. An extended experimental evaluation shows that the reactive posture behaviors are able to create an effective and efficient locomotion in challenging environments.

As intelligent vehicles become more and more capable, they must learn to navigate and localize themselves in a wide variety of environments, including GPS-denied and only crudely mapped areas. We argue that since autonomous vehicles must be able to perceive, and semantically interpret, their immediate environment, they should be able to use abstract semantic information as their sole means of localization. This simplifies the level of detail and precision required from environment maps so that, for example, a rough floor plan of a parking garage will suffice to autonomously navigate it. We propose a concept for semantic localization which only requires a conceptual semantic map of the environment, and can be made to work with any kind of sensor data from which the required semantic information can be extracted. We present a localization algorithm which may be used as a base for semantic navigation, e.g. in context of automated driving, and some initial results of its application in a parking garage scenario.

3D data collected by a laser scanner has great potential for robotic applications. Exact geometrical models of the environment surrounding the robot can be created from these point clouds. But, before creating any model, the 3D point cloud has to be segmented and depending on the size and quality of the point cloud, this can be a very challenging task. This article describes a robust 3D scan segmentation technique, which is capable of segmenting a 3D point cloud in a short amount of time. The results of the segmentation are used to assist a teleoperator to manoeuvre a robot through an unknown environment. Our segmentation approach copes with indoor and outdoor environments, using only a minimum of assumptions, which makes it very robust. A 3D visualisation illustrates the segmentation results in a clear and user-friendly way.