Combining system knowledge with deep learning for fault diagnosis in industrial applications offers the potential to reduce the dependency of deep learning algorithms on extensive labeled datasets. However, existing methods often rely on highly specialized, problem-specific knowledge or demand detailed physical insights into the system, which limits their generalizability. Additionally, inconsistencies in knowledge representation hinder the ability to compare and build upon prior approaches. In this work, we address these challenges by leveraging commonly available knowledge about the phase structure of systems and the hierarchical organization of condition spaces. This information is systematically represented using knowledge graphs (KGs) based on the Resource Description Framework (RDF). To integrate this knowledge into deep learning, we transform the input data and the corresponding labels based on the KGs, and employ a graph neural network (GNN) trained with a semantic loss function informed by the knowledge about the condition space. The proposed approach is evaluated on three diverse datasets with varying characteristics under the two scenarios of domain generalization and novel fault detection.

Due to the growing connectivity capabilities of manufacturing equipment, an increasing number of machines and systems are being integrated into corporate networks, where they continuously transmit data. As manufacturing processes become more complex, they generate extensive volumes of domain-specific data, necessitating intelligent and adaptive data processing solutions. AI applications in manufacturing need to be developed to effectively process this data. In recent years, Large Language Models (LLMs) have received considerable attention, particularly in fields such as software engineering, data engineering, and computer simulation. However, their application in manufacturing contexts remains in its infancy, both in industrial practice and academic research. This paper explores approaches to leveraging textual data within manufacturing, specifically by comparing traditional Natural Language Processing (NLP) pipelines with LLM-based pre-processing techniques. Initially, a classical NLP pipeline approach is analyzed to establish a baseline. Subsequently, an LLM-driven method aimed at improving contextual processing is examined. The findings present a simplified and practical framework for employing LLMs in industrial data science projects. Our results indicate that using LLMs for various data pre-processing tasks significantly enhances machine learning performance metrics while concurrently reducing the effort and the level of data science expertise required.

Accurate uncertainty quantification is critical for robust and trustworthy predictions in many real-world applications. Bayesian Neural Networks (BNNs) provide a principled approach for modeling uncertainty but are often limited by the computational complexity of Bayesian inference. In this paper, we introduce a statistical linearization approach for multilayer feedforward BNNs. We demonstrate that this statistical linearization is equivalent to the Kalman Bayesian Neural Networks (KBNN) framework. This equivalence unifies these methodologies, providing a theoretical foundation for understanding the relationship between different BNN training approaches.

To gain deeper insights into a complex sensor system through the lens of causality, we present common and individual causal mechanism estimation (CICME), a novel threestep approach to inferring causal mechanisms from heterogeneous data collected across multiple domains. By leveraging the principle of Causal Transfer Learning (CTL), CICME is able to reliably detect domain-invariant causal mechanisms when provided with sufficient samples. The identified common causal mechanisms are further used to guide the estimation of the remaining causal mechanisms in each domain individually. The performance of CICME is evaluated on linear Gaussian models under scenarios inspired from a manufacturing process. Building upon existing continuous optimization-based causal discovery methods, we show that CICME leverages the benefits of applying causal discovery on the pooled data and repeatedly on data from individual domains, and it even outperforms both baseline methods under certain scenarios.

Predicting future video frames is a challenging task with many downstream applications. Previous work [1], [2] has shown that procedural knowledge enables deep models for complex dynamical settings, however their model ViPro assumed a given ground truth initial symbolic state. We show that this approach led to the model learning a shortcut that does not actually connect the observed environment with the predicted symbolic state, resulting in the inability to estimate states given an observation if previous states are noisy. In this work, we add several improvements to ViPro that enables the model to correctly infer states from observations without providing a full ground truth state in the beginning. We show that this is possible in an unsupervised manner, and extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.

Although machine learning (ML) has been widely utilized across industries, its integration into conformity assessment processes is still a key challenge. This paper proposes a framework that enables ML systems to replace manual and destructive testing by directly conducting conformity assessments based on process parameters. The framework aligns with standards like Verband der Automobilindustrie (VDA) Volume 5 to support the development of ML-based conformity assessment systems. To enable this integration, gaps in process integration, role definitions, and technical validation are identified through a gap analysis. The framework addresses process- and role-related gaps by adapting each step of the VDA Volume 5 development process—planning, validation, calibration, and deployment—to meet conformity requirements. Developed during the planning phase at a German automotive manufacturer, this framework provides a structured approach for designing ML systems that comply with industry standards, contributing to more efficient and reliable quality assurance (QA) processes.

Robots operating in everyday environments encounter a wide variety of previously unseen objects. Deep Learning methods simplify unknown object and scene segmentation by structuring inherent real-world complexities, improving visual scene understanding. However, they need vast amounts of labeled high-variance data for training. Acquiring these labels for rich real-world data requires significant manual effort, especially for segmentation masks. Although interactive segmentation accelerates this process, these methods still require substantial manual interaction, and the creation of large datasets remains labor-intensive. Consequently, there is a lack of diverse, high-quality datasets for unknown object instance segmentation in everyday environments. This research proposes a semi-automatic, RGB-only algorithmic pipeline for annotating novel objects, reducing manual effort to iteratively placing objects in the scene. We investigate several change detection-based approaches, including remote sensing change detection methods (TTP model), the DeepBackgroundMattingV2 image matting model, and the Segment Anything Model (SAM1 + SAM2) prompted with automatically extracted change regions. We propose the novel ILIS dataset to evaluate these methods in challenging everyday scenes, displaying reliable automatic mask proposal performance of up to 0.9549 mIoU and 0.9565 boundary F1 score. This highlights the potential of this method to accelerate large-scale dataset creation, saving at least 27.27 hours per 1,000 images by eliminating manual annotations.

In layout-to-image (L2I) synthesis, controlled complex scenes are generated from coarse information like bounding boxes. Such a task is exciting to many downstream applications because the input layouts offer strong guidance to the generation process while remaining easily reconfigurable by humans. In this paper, we proposed STyled LAYout Diffusion (STAY Diffusion), a diffusion-based model that produces photo-realistic images and provides fine-grained control of stylized objects in scenes. Our approach learns a global condition for each layout, and a self-supervised semantic map for weight modulation using a novel Edge-Aware Normalization (EA Norm). A new Styled-Mask Attention (SM Attention) is also introduced to cross-condition the global condition and image feature for capturing the objects' relationships. These measures provide consistent guidance through the model, enabling more accurate and controllable image generation. Extensive benchmarking demonstrates that our STAY Diffusion presents high-quality images while surpassing previous state-of-the-art methods in generation diversity, accuracy, and controllability.

This paper investigates the relationship between law and eXplainable Artificial Intelligence (XAI). While there is much discussion about the AI Act, which was adopted by the European Parliament in March 2024, other areas of law seem underexplored. This paper focuses on European (and in part German) law, although with international concepts and regulations such as fiduciary duties, the General Data Protection Regulation (GDPR), and product safety and liability. Based on XAI-taxonomies, requirements for XAI methods are derived from each of the legal fields, resulting in the conclusion that each legal field requires different XAI properties and that the current state of the art does not fulfill these to full satisfaction, especially regarding the correctness (sometimes called fidelity) and confidence estimates of XAI methods.

Machinery fault diagnosis is increasingly reliant on data-driven algorithms, yet struggles with adapting to unseen operating conditions. To address this, we propose integrating phase information into heterogeneous graphs for fault diagnostics with Graph Neural Networks (GNNs). Our method involves identifying the distinct phases that a machine undergoes within a cycle and segmenting the signals accordingly. These segmented signals are then represented in a graph of multiple connected sensor networks with diverse node and edge types. Prior to graph classification with a GNN, individual Convolutional Neural Networks (CNNs) preprocess the node attributes to account for their unique characteristics. The evaluation in a domain adaptation setting demonstrates the effectiveness of our approach, offering insights into improving the robustness and domain adaptability of fault diagnosis models.

Many works have been devoted to evaluating the robustness of a classifier in the neighborhood of single points of input data. Recently, in particular, probabilistic settings have been considered, where robustness is defined in terms of random perturbations of input data. In this paper, we consider robustness on the entire input domain as opposed to single points of input. For the first time, we provide formal guarantees on the probability of robustness, given a random input and a random perturbation, based only on sampling or in combination with existing pointwise methods. We prove that the error becomes arbitrarily small for enough input data. This is applicable to any classification or regression model and any random input perturbation. We then illustrate the resulting bounds and compare them against the state of the art for models trained on the MNIST, California Housing, and ImageNet datasets.

Bayesian Neural Networks (BNNs) offer a sophisticated framework for extending classical neural network point estimates to encompass predictive distributions. Despite the high potential of BNNs, established BNN training methods such as Variational Inference (VI) and Markov Chain Monte Carlo (MCMC) grapple with issues such as scalability and hyperparameter dependence. In addressing these issues, our research focuses on the fundamental elements of BNNs, in particular perceptrons and their predictive capabilities. We introduce a new perspective on the closed-form solution for backward-pass computation for the Bayesian perceptron and prove that the state-of-the-art solution is equivalent to statistical linearization. To assess the efficacy of Bayesian perceptrons and provide insights into their performance in distinct input space regions, a novel methodology utilizing k-d trees as a space partitioning method is introduced to evaluate prediction quality within specific input space regions.

Data fusion deals with combining information from multiple sensors to support decision making. In such settings, machine learning methods, that principally only take correlation into account, have been applied widely due to their strong predictive and computational capabilities. In this paper, we investigate potential benefits of introducing causal knowledge in machine learning-based data fusion to address two common downstream tasks, namely, quality prediction and root cause analysis (RCA). To resemble the complex relationships typically associated with sensor data, we create simulation data with explicit modeling of latent confounding. The results of this study indicate that taking into account true causal knowledge significantly improves the performance of RCA, and leads to prediction models that are more robust to severe distribution shifts in the presence of latent confounding. Furthermore, if causal knowledge needs to be inferred from observational data using existing causal discovery methods, we propose a selection criterion to choose the best causal structure. We show that given a sufficient amount of data, the selected causal structure can be used as reliable input to solve the downstream tasks.

Surface treatment tasks such as grinding, sanding or polishing are a vital step of the value chain in many industries, but are notoriously challenging to automate. We present RoboGrind, an integrated system for the intuitive, interactive automation of surface treatment tasks with industrial robots. It combines a sophisticated 3D perception pipeline for surface scanning and automatic defect identification, an interactive voice-controlled wizard system for the AI-assisted bootstrapping and parameterization of robot programs, and an automatic planning and execution pipeline for force-controlled robotic surface treatment. RoboGrind is evaluated both under laboratory and real-world conditions in the context of refabricating fiberglass wind turbine blades.

The identification of a mathematical dynamics model is a crucial step in the designing process of a controller. However, it is often very difficult to identify the system’s governing equations, especially in complex environments that combine physical laws of different disciplines. In this paper, we present a new approach that allows identifying an ordinary differential equation by means of a physics-informed machine learning algorithm. Our method introduces a special neural network that allows exploiting prior human knowledge to a certain degree and extends it autonomously, so that the resulting differential equations describe the system as accurately as possible. We validate the method on a Duffing oscillator with simulation data and, additionally, on a cascaded tank example with real-world data. Subsequently, we use the developed algorithm in a model-based reinforcement learning framework by alternately identifying and controlling a system to a target state. We test the performance by swinging-up an inverted pendulum on a cart.

To this day, only a small number of industrial robots is used in assembly. One key reason for this is that specific contact situations require the introduction of force-control schemes. The parameters for those schemes are hard to select in practice, because they require in-depth expertise about the robot and the process. Learning-from-Demonstration (LfD) provides a powerful approach to intuitively parameterize robot programs by demonstrating the task at hand. However, when dimensions increase by including force or orientation, many LfD algorithms are hard to verify, understand and maintain, requiring expert knowledge to make adaptions, effectively making it a "black-box". This property renders them ineffective for usage in industrial applications. We build upon a system of composable skills, that can be easily adapted by experts without the need to demonstrate the task again. This approach to skill-based robot programming promises to address the issues of readability and maintainability by sequencing robot movements in skills and breaking them down into understandable (sub-)goals. In this paper, we combine skill-based programming with LfD, preserving both maintainability and intuitive parameterization. We present (a) an approach to parameterize and create sequences of hierarchies of force- and/or position-controlled robot skills from a LfD model, (b) which can be adapted by a user by hand with few, basic and understandable parameters, and (c) show its applicability on the real-world example of terminal clamp assembly. We achieve a reduction in teach-in time of 53.8% for variants, increased robustness against variance, and efficient tight stacking of clamps with a gap of ≤ 1mm.

Recent deep generative models (DGMs) such as generative adversarial networks (GANs) and diffusion probabilistic models (DPMs) have shown their impressive ability in generating high-fidelity photorealistic images. Although looking appealing to human eyes, training a model on purely synthetic images for downstream image processing tasks like image classification often results in an undesired performance drop compared to training on real data. Previous works have demonstrated that enhancing a real dataset with synthetic images from DGMs can be beneficial. However, the improvements were subjected to certain circumstances and yet were not comparable to adding the same number of real images. In this work, we propose a new taxonomy to describe factors contributing to this commonly observed phenomenon and investigate it on the popular CIFAR-10 dataset. We hypothesize that the Content Gap accounts for a large portion of the performance drop when using synthetic images from DGM and propose strategies to better utilize them in downstream tasks. Extensive experiments on multiple datasets showcase that our method outperforms baselines on downstream classification tasks both in case of training on synthetic only (Synthetic-to-Real) and training on a mix of real and synthetic data (Data Augmentation), particularly in the data-scarce scenario.

Taking over arbitrary tasks like humans do with a mobile service robot in open-world settings requires a holistic scene perception for decision-making and high-level control. This article presents a human-inspired scene perception model to minimize the gap between human and robotic capabilities. The approach takes over fundamental neuroscience concepts, such as a triplet perception split into recognition, knowledge representation, and knowledge interpretation. A recognition system splits the background and foreground to integrate exchangeable image-based object detectors and simultaneous localization and mapping, a multilayer knowledge base represents scene information in a hierarchical structure and offers interfaces for high-level control, and knowledge interpretation methods deploy spatio-temporal scene analysis and perceptual learning for self-adjustment. A single-setting ablation study is used to evaluate the impact of each component on the overall performance for a fetch-and-carry scenario in two simulated and one real-world environment.

Compared to point estimates calculated by standard neural networks, Bayesian neural networks (BNN) provide probability distributions over the output predictions and model parameters, i.e., the weights. Training the weight distribution of a BNN, however, is more involved due to the intractability of the underlying Bayesian inference problem and thus, requires efficient approximations. In this paper, we propose a novel approach for BNN learning via closed-form Bayesian inference. For this purpose, the calculation of the predictive distribution of the output and the update of the weight distribution are treated as Bayesian filtering and smoothing problems, where the weights are modeled as Gaussian random variables. This allows closed-form expressions for training the network's parameters in a sequential/online fashion without gradient descent. We demonstrate our method on several UCI datasets and compare it to the state of the art.

Recent advances in Reinforcement Learning (RL) have made significant contributions in past years by offering intelligent solutions to solve robotic tasks. However, most RL algorithms, especially the model-free RL, are plagued by low learning efficiency and safety problems. In this paper, we propose using the Bayesian Neural Networks (BNNs) to guide the agent exploring actively to enhance the learning efficiency in RL and investigate the potential of recognizing safety risks in working environments with uncertainty information. We compare two types of uncertainty quantification methods in both action and state spaces. To validate our method, we visualize the quantified uncertainty in robot environments with or without safety hazards. Moreover, we evaluate the learning efficiency and safety performance of the RL agents learned with BNNs on different robotic tasks.

Laser cutting is one of the classic methods used in metal processing. With increasing automation, it is important to ensure that large volumes can be produced reliably. This includes avoiding re-welding, known as cut interruption. In the presented work, audio signals are used to detect cut interruptions during laser cutting. The audio signal is classified into two classes: good cuts and cut interruptions. To solve this classification problem, the time series classifier RandOm Convolutional KErnel Transform (ROCKET) is used. The influence of the window size, the number of kernels and the repeatability of the training is investigated. With the presented work it is shown that a cut interruption detection with a microphone is possible. For a real world application there is a trade-off between accuracy and window size.

Robotic applications like automated order picking in warehouses or retail stores, or fetch and carry tasks in hospitals, care homes, or households rely on the capability of service robots to find and handle a specific type of object. These applications are challenging as the set of objects is very large and varies over time. Despite its significance, there is no suitable universal large-scale dataset available from the retail domain, which allows for a principled analysis of all relevant robotics research aspects in that field. Hence, this paper introduces a novel dataset of more than 1,000 retail objects, including color images, 3D scans, and high-resolution textured 3D models of individual objects, synthetic scenes and real settings, which covers the specifics of the retail domain. The dataset was designed to serve researchers in all relevant robotics tasks in retail like 3D reconstruction and object modeling, large-scale object classification and instance detection including incremental learning and fine-grained detection, text reading, logo detection, semantic grounding and affordance detection, grasp analysis and manipulation planning, as well as digital twinning and virtual environments. Based on synthetic RGB images of scenes created from the 3D models, two exemplary use cases are examined in this paper to demonstrate the benefits of the dataset: we evaluate the state-of-the-art incremental object detection method InstanceNet and a few-shot fine-grained object classification method. The results prove the suitability of InstanceNet for incremental object detection on large datasets and are promising for the few-shot object classification system.

In recent years, the development of Artificial Intelligence (AI) has shown tremendous potential in diverse areas. Among them, reinforcement learning (RL) has proven to be an effective solution for learning intelligent control strategies. As an inevitable trend for mitigating climate change, hybrid electric vehicles (HEVs) rely on efficient energy management strategies (EMS) to minimize energy consumption. Many researchers have employed RL to learn optimal EMS for specific vehicle models. However, most of these models tend to be complex and proprietary, making them unsuitable for broad applicability. This paper presents a novel framework, in which we implement and integrate RL-based EMS with the open-source vehicle simulation tool called FASTSim. The learned RL-based EMSs are evaluated on various vehicle models using different test drive cycles and prove to be effective in improving energy efficiency.

Manual assembly workers face increasing complexity in their work. Human-centered assistance systems could help, but object recognition as an enabling technology hinders a sophisticated human-centered design of these systems. At the same time, activity recognition based on hand poses suffers from poor pose estimation in complex usage scenarios, such as wearing gloves. This paper presents a self-supervised pipeline for adapting hand pose estimation to specific use cases with minimal human interaction. This enables cheap and robust hand pose-based activity recognition. The pipeline consists of a general machine learning model for hand pose estimation trained on a generalized dataset, spatial and temporal filtering to account for anatomical constraints of the hand, and a retraining step to improve the model. Different parameter combinations are evaluated on a publicly available and annotated dataset. The best parameter and model combination is then applied to unlabeled videos from a manual assembly scenario. The effectiveness of the pipeline is demonstrated by training an activity recognition as a downstream task in the manual assembly scenario.

Being able to predict the remaining useful life (RUL) of an engineering system is an important task in prognostics and health management. Recently, data-driven approaches to RUL predictions are becoming prevalent over model-based approaches since no underlying physical knowledge of the engineering system is required. Yet, this just replaces required expertise of the underlying physics with machine learning (ML) expertise, which is often also not available. Automated machine learning (AutoML) promises to build end-to-end ML pipelines automatically enabling domain experts without ML expertise to create their own models. This paper introduces AutoRUL, an AutoML-driven end-to-end approach for automatic RUL predictions. AutoRUL combines fine-tuned standard regression methods to an ensemble with high predictive power. By evaluating the proposed method on eight real-world and synthetic datasets against state-of-the-art hand-crafted models, we show that AutoML provides a viable alternative to hand-crafted data-driven RUL predictions. Consequently, creating RUL predictions can be made more accessible for domain experts using AutoML by eliminating ML expertise from data-driven model construction.

In the last 10 years, various automated machine learning (AutoML) systems have been proposed to build end-to-end machine learning (ML) pipelines with minimal human interaction. Even though such automatically synthesized ML pipelines are able to achieve competitive performance, recent studies have shown that users do not trust models constructed by AutoML due to missing transparency of AutoML systems and missing explanations for the constructed ML pipelines. In a requirements analysis study with 36 domain experts, data scientists, and AutoML researchers from different professions with vastly different expertise in ML, we collect detailed informational needs for AutoML. We propose XAutoML , an interactive visual analytics tool for explaining arbitrary AutoML optimization procedures and ML pipelines constructed by AutoML. XAutoML combines interactive visualizations with established techniques from explainable artificial intelligence (XAI) to make the complete AutoML procedure transparent and explainable. By integrating XAutoML with JupyterLab , experienced users can extend the visual analytics with ad-hoc visualizations based on information extracted from XAutoML . We validate our approach in a user study with the same diverse user group from the requirements analysis. All participants were able to extract useful information from XAutoML , leading to a significantly increased understanding of ML pipelines produced by AutoML and the AutoML optimization itself.

In manufacturing, the current trend-shift from mass-production to mass-personalization is enabled, among others, by the emerging field of human-robot collaboration (HRC), in which humans collaborate or work in proximity with robots. In HRC scenarios, robots need to exert a desired behaviour that maximizes utility without sacrificing safety and responsiveness. To maximize safety and utility in static environments, state-of-the-art offline motion-planners use computationally-heavy algorithms for approximating the collision-free robot reachability and accordingly generate (sub-)optimal robot trajectories. To enable real-time responsiveness, we propose an integrated global planner to generate sub-optimal trajectories. It relies on a closed-loop reactive controller for executing the global plan while ensuring safety with practical assumptions about the environment. We evaluate GLIR in simulation. In our experiments, our global planner operates at 25 Hz and the local planner at 100 Hz, enabling their execution in dynamic environments. In all experiments on static scenes with static and dynamic goals, GLIR keeps a safety distance from obstacles. We showcase some simulation experiments and a real-world demonstration in the video available at https://mohamedgalil.github.io/glir/.

This work introduces a novel interpretable machine learning method called Mixture of Decision Trees (MoDT). It constitutes a special case of the Mixture of Experts ensemble architecture, which utilizes a linear model as gating function and decision trees as experts. Our proposed method is ideally suited for problems that cannot be satisfactorily learned by a single decision tree, but which can alternatively be divided into subproblems. Each subproblem can then be learned well from a single decision tree. Therefore, MoDT can be considered as a method that improves performance while maintaining interpretability by making each of its decisions understandable and traceable to humans.Our work is accompanied by a Python implementation, which uses an interpretable gating function, a fast learning algorithm, and a direct interface to fine-tuned interpretable visualization methods. The experiments confirm that the implementation works and, more importantly, show the superiority of our approach compared to single decision trees and random forests of similar complexity.

Increasingly volatile markets challenge companies and demand flexible production systems that can be quickly adapted to new conditions. Machine Learning has proven to show significant potential in supporting the human operator during the time-consuming and complex task of robot pro-gramming by identifying relevant parameters of the underlying robot control program. We present a solution to learn these parameters for contact-rich, force-controlled assembly tasks from a simulation using hardware-independent robot skills. We show that successful learning and real-world execution are possible even under process deviation and tolerances utilizing the designed learning system. We present learning skill param-eters as high-level robot control, evaluation and comparison of extensive simulations, and preliminary experiments on a physical robot test-bed. The developed solution approach is evaluated and discussed using the Peg-in-Hole process, a typical benchmark process in force-controlled assembly.

In this paper, we present a Domain Randomization and a Domain Adaptation approach to transfer experience for entanglement detection and separation from simulation into a real-world bin-picking application. We investigate the influence of different randomization options in image processing and use a CycleGAN as a further Domain Adaptation method to synthesize simulation data as realistically as possible. On the basis of this adapted data we re-train our detection and separation methods and validate the usefulness of these Sim-to-Real methods. In numerous real-world experiments we show that we achieve a significant increase of up to 71.74 % in the performance of the overall system by using the Sim-to-Real approaches as opposed to the direct transfer.

In highly competitive markets, customer loyalty plays an increasingly important role for companies. An important aspect is the recording and processing of customer complaints, on the one hand for problem-solving and on the other hand for internal process optimization. The complaint handling process extends over several phases, in which different people with varying expertise and experience may be involved, which poses a major challenge in order to achieve Consistent quality in the recording process of complaints. To address this issue, this work presents a robust active learning based AI system that allows using existing expert knowledge to create more reliable classifications in a shorter amount of time. The implemented prototype shows a decrease of up to 86% in the average processing time of complaints, with an average increase of up to 37% in the classification accuracy.

Single shot approaches have demonstrated tremendous success on various computer vision tasks. Finding good parameterizations for 6D object pose estimation remains an open challenge. In this work, we propose different novel parameterizations for the output of the neural network for single shot 6D object pose estimation. Our learning-based approach achieves state-of-the-art performance on two public benchmark datasets. Furthermore, we demonstrate that the pose estimates can be used for real-world robotic grasping tasks without additional ICP refinement.

Maintenance scheduling is a complex decision-making problem in the production domain, where a number of maintenance tasks and resources has to be assigned and scheduled to production entities in order to prevent unplanned production downtime. Intelligent maintenance strategies are required that are able to adapt to the dynamics and different conditions of production systems. The paper introduces a deep reinforcement learning approach for condition-oriented maintenance scheduling in flow line systems. Different policies are learned, analyzed and evaluated against a benchmark scheduling heuristic based on reward modelling. The evaluation of the learned policies shows that reinforcement learning based maintenance strategies meet the requirements of the presented use case and are suitable for maintenance scheduling in the shop floor.

This paper introduces a novel approach for the grasping and precise placement of various known rigid objects using multiple grippers within highly cluttered scenes. Using a single depth image of the scene, our method estimates multiple 6D object poses together with an object class, a pose distance for object pose estimation, and a pose distance from a target pose for object placement for each automatically obtained grasp pose with a single forward pass of a neural network.By incorporating model knowledge into the system, our approach has higher success rates for grasping than state-of-the-art model-free approaches. Furthermore, our method chooses grasps that result in significantly more precise object placements than prior model-based work.

Programming-by-Demonstration (PbD) or Imitation Learning (IL) provides a powerful approach to program robots intuitively. In industrial settings, these approaches are not commonly deployed since several factors negatively impact their productive use. The most common reasons are safety concerns on various levels. First, industrial robots are only allowed to be operated in direct contact if the operator has sufficient experience. Secondly, many PbD systems do not incorporate force measurements in their model directly. This renders them ineffective for assembly tasks, such as snap- fit connections. In this paper we (a) present an approach to incorporate force measurements into a generative model (b) using only external sensors without relying on measurements from the robot during demonstrating to decouple the teaching process from the robot and (c) show the benefit of explicit force measurement and modeling on a Franka Emika Panda robot by the example of assembling terminal clamps on a DIN rail.

Abstract With the rapid improvements in machine learning and deep learning, decisions made by automated decision support systems (DSS) will increase. Besides the accuracy of predictions, their explainability becomes more important. The algorithms can construct complex mathematical prediction models. This causes insecurity to the predictions. The insecurity rises the need for equipping the algorithms with explanations. To examine how users trust automated DSS, an experiment was conducted. Our research aim is to examine how participants supported by an DSS revise their initial prediction by four varying approaches (treatments) in a between‐subject design study. The four treatments differ in the degree of explainability to understand the predictions of the system. First we used an interpretable regression model, second a Random Forest (considered to be a black box [BB]), third the BB with a local explanation and last the BB with a global explanation. We noticed that all participants improved their predictions after receiving an advice whether it was a complete BB or an BB with an explanation. The major finding was that interpretable models were not incorporated more in the decision process than BB models or BB models with explanations.

Predictions obtained by, e.g., artificial neural networks have a high accuracy but humans often perceive the models as black boxes. Insights about the decision making are mostly opaque for humans. Particularly understanding the decision making in highly sensitive areas such as healthcare or finance, is of paramount importance. The decision-making behind the black boxes requires it to be more transparent, accountable, and understandable for humans. This survey paper provides essential definitions, an overview of the different principles and methodologies of explainable Supervised Machine Learning (SML). We conduct a state-of-the-art survey that reviews past and recent explainable SML approaches and classifies them according to the introduced definitions. Finally, we illustrate principles by means of an explanatory case study and discuss important future directions.

Machine learning (ML) has become a vital part in many aspects of our daily life. However, building well performing machine learning applications requires highly specialized data scientists and domain experts. Automated machine learning (AutoML) aims to reduce the demand for data scientists by enabling domain experts to build machine learning applications automatically without extensive knowledge of statistics and machine learning. This paper is a combination of a survey on current AutoML methods and a benchmark of popular AutoML frameworks on real data sets. Driven by the selected frameworks for evaluation, we summarize and review important AutoML techniques and methods concerning every step in building an ML pipeline. The selected AutoML frameworks are evaluated on 137 data sets from established AutoML benchmark suites.

Safety in Human-Robot Collaboration (HRC) is a bottleneck to HRC-productivity in industry. With robots being the main source of hazards, safety engineers use over-emphasized safety measures, and carry out lengthy and expensive risk assessment processes on each HRC-layout reconfiguration. Recent advances in deep Reinforcement Learning (RL) offer solutions to add intelligence and comprehensibility of the environment to robots. In this paper, we propose a framework that uses deep RL as an enabling technology to enhance intelligence and safety of the robots in HRC scenarios and, thus, reduce hazards incurred by the robots. The framework offers a systematic methodology to encode the task and safety requirements and context of applicability into RL settings. The framework also considers core components, such as behavior explainer and verifier, which aim for transferring learned behaviors from research labs to industry. In the evaluations, the proposed framework shows the capability of deep RL agents learning collision-free point-to-point motion on different robots inside simulation, as shown in the supplementary video.

In this paper, we introduce a novel single shot approach for 6D object pose estimation of rigid objects based on depth images. For this purpose, a fully convolutional neural network is employed, where the 3D input data is spatially discretized and pose estimation is considered as a regression task that is solved locally on the resulting volume elements. With 65 fps on a GPU, our Object Pose Network (OP-Net) is extremely fast, is optimized end-to-end, and estimates the 6D pose of multiple objects in the image simultaneously. Our approach does not require manually 6D pose-annotated real-world datasets and transfers to the real world, although being entirely trained on synthetic data. The proposed method is evaluated on public benchmark datasets, where we can demonstrate that state-of-the-art methods are significantly outperformed.

In this paper, we introduce a novel learning-based approach for grasping known rigid objects in highly cluttered scenes and precisely placing them based on depth images. Our Placement Quality Network (PQ-Net) estimates the object pose and the quality for each automatically generated grasp pose for multiple objects simultaneously at 92 fps in a single forward pass of a neural network. All grasping and placement trials are executed in a physics simulation and the gained experience is transferred to the real world using domain randomization. We demonstrate that our policy successfully transfers to the real world. PQ-Net outperforms other model-free approaches in terms of grasping success rate and automatically scales to new objects of arbitrary symmetry without any human intervention.

Remarkable progress in the field of machine learning strongly drives the research in many application domains. For some domains, it is mandatory that the output of machine learning algorithms needs to be interpretable. In this paper, we propose a rule-based regularization technique to enforce interpretability for neural networks (NN). For this purpose, we train a rule-based surrogate model simultaneously with the NN. From the surrogate, a metric quantifying its degree of explainability is derived and fed back to the training of the NN as a regularization term. We evaluate our model on four datasets and compare it to unregularized models as well as a decision tree (DT) based baseline. The rule-based regularization approach achieves interpretability and competitive accuracy.

One obstacle that so far prevents the introduction of machine learning models primarily in critical areas is the lack of explainability. In this work, a practicable approach of gaining explainability of deep artificial neural networks (NN) using an interpretable surrogate model based on decision trees is presented. Simply fitting a decision tree to a trained NN usually leads to unsatisfactory results in terms of accuracy and fidelity. Using L1-orthogonal regularization during training, however, preserves the accuracy of the NN, while it can be closely approximated by small decision trees. Tests with different data sets confirm that L1-orthogonal regularization yields models of lower complexity and at the same time higher fidelity compared to other regularizers.

In this paper, we introduce a new public dataset for 6D object pose estimation and instance segmentation for industrial bin-picking. The dataset comprises both synthetic and real-world scenes. For both, point clouds, depth images, and annotations comprising the 6D pose (position and orientation), a visibility score, and a segmentation mask for each object are provided. Along with the raw data, a method for precisely annotating real-world scenes is proposed.To the best of our knowledge, this is the first public dataset for 6D object pose estimation and instance segmentation for bin-picking containing sufficiently annotated data for learning-based approaches. Furthermore, it is one of the largest public datasets for object pose estimation in general. The dataset is publicly available at http://www.bin-picking.ai/en/ dataset.html.

In a surveillance environment with high clutter, finding the correct measurement to track associations becomes extremely important for efficient target tracking. This study offers a novel algorithm to retrodict the data association probabilities at any past time instant, when the batch set of measurements is kept in memory. For the retrodiction procedure, the batch association cost is first written explicitly as a binary integer optimization problem with a quadratic cost function and it is shown that the relaxed form of the problem is convex. From the relaxed problem, a lower bound for the optimal association cost is derived, and this lower bound is used as the data association probabilities pertaining to that selected time instant in the past. Due to its consideration of the batch set of data in a retrospective manner, we will call this algorithm as Retrodictive Probabilistic Data Association, RPDA. For simplification of the mathematical analysis, a single point target with no missing measurements, i.e. PD= 1, is taken into account.

This Special Section was inspired by the 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI 2016), see mfi2016.org, which took place 19–21 September 2016 in Baden-Baden, Germany. The conference was sponsored by the IEEE Robotics and Automation Society (RAS) and the IEEE Industrial Electronics Society (IES). Several of the papers in this Special Section build upon papers presented at the conference. While a total of 106 papers were presented at MFI 2016, we invited only the authors of the highest-ranked papers to submit an extended version of their respective conference papers to this Special Section. Besides contacting these selected authors, an open call for papers for this Special Session was published to broaden the scope and to allow for papers not presented at MFI 2106. For extended conference papers, an important constraint was to include substantially novel aspects, such as added theoretical work or new experimental results. Authors are required to cite their original conference papers and clearly specify the novel contribution of the journal paper with respect to the conference paper. All papers, those building upon previously presented MFI 2016 papers and additional papers submitted related to the open call for papers, underwent the same rigorous review process as regular submissions strictly according to the guidelines of the IEEE Transactions on Industrial Informatics (TII). A total of 58 papers were submitted to this Special Section of the IEEE TII. After a careful review process, eight papers were finally selected for publication. We have grouped the eight accepted papers into four sections. The papers are briefly summarized here.

Event correlation is the task of detecting dependencies between events in event sequences, e.g., for predictive maintenance based on log-files. In this work, a new data-driven, generic framework for event correlation is presented. First, we use a fast preliminary test statistic to determine candidate event type pairs. Next, the precise distribution of the time lag between those pairs is calculated. For this purpose, a new efficient iterative method is developed that aligns two event sequences and finds the optimal event assignments. In our experiments, the proposed method is orders of magnitude faster than state-of-the-art methods but always yields similar (or even better) results.

In this paper, a framework for collaborative face recognition from video sequences in a multi-camera environment is proposed. Collaboration between cameras allows for higher recognition performance in both the common and non-common field-of-view (FOV) cases. For the latter, the appearance of an object in a nearby camera is predicted using the last tracked position of the object paired with a time-of-arrival model between camera pairs. An experiment using four cameras in an office environment confirms the applicability and performance gains of the proposed framework.

The number of smart things is growing exponentially. By 2020, tens of billions of things will be deployed worldwide, collecting a wealth of diverse data. Traditional computing models collect in-field data and then transmit it to a central data center where analytics are applied to it, but this is no longer a sustainable model. New approaches and new technologies are required to transform enormous amounts of collected data into meaningful information. Technology also will enable the interconnection around things in the IoT ecosystem but further research is required in the development, convergence and interoperability of the different IoT elements. In this paper, we provide a picture of the main technological components needed to enable the interconnection among things in order to realize IoT concepts and applications.

For large data sets, performing Gaussian process regression is computationally demanding or even intractable. If data can be processed sequentially, the recursive regression method proposed in this paper allows incorporating new data with constant computation time. For this purpose two operations are performed alternating on a fixed set of so-called basis vectors used for estimating the latent function: First, inference of the latent function at the new inputs. Second, utilization of the new data for updating the estimate. Numerical simulations show that the proposed approach significantly reduces the computation time and at the same time provides more accurate estimates compared to existing on-line and/or sparse Gaussian process regression approaches.

In range-based localization, the trajectory of a mobile object is estimated based on noisy range measurements between the object and known landmarks. In order to deal with this uncertain information, a Bayesian state estimator is presented, which exploits optimal stochastic linearization. Compared to standard state estimators like the Extended or Unscented Kalman Filter, where a point-based Gaussian approximation is used, the proposed approach considers the entire Gaussian density for linearization. By employing the common assumption that the state and measurements are jointly Gaussian, the linearization can be calculated in closed form and thus analytic expressions for the range-based localization problem can be derived.

An algorithm for estimating the pose, i.e., translation and rotation, of an extended target object is introduced. Compared to conventional methods, where pose estimation is performed on the basis of time-of-flight (TOF) measurements between external sources and sensors attached to the object, the proposed approach directly uses the amplitude values measured at the sensors for estimation purposes without an intermediate TOF estimation step. This is achieved by modeling the wave propagation by a nonlinear dynamic system comprising a system and a measurement equation. The nonlinear system equation includes a model of the time-variant structure of the object rotation based on rotation vectors. As a result, the measured amplitude values at the sensors can be processed instantaneously in a recursive fashion. Uncertainties in the measurement process are systematically considered by employing a stochastic filter for estimating the pose, i.e., the state of the nonlinear dynamic system.

We propose an analytic moment-based filter for nonlinear stochastic dynamic systems modeled by Gaussian processes. Exact expressions for the expected value and the covariance matrix are provided for both the prediction step and the filter step, where an additional Gaussian assumption is exploited in the latter case. Our filter does not require further approximations. In particular, it avoids finite-sample approximations. We compare the filter to a variety of Gaussian filters, that is, the EKF, the UKF, and the recent GP-UKF proposed by Ko et al. (2007).

In nonlinear Bayesian estimation it is generally inevitable to incorporate approximate descriptions of the exact estimation algorithm. There are two possible ways to involve approximations: Approximating the nonlinear stochastic system model or approximating the prior probability density function. The key idea of the introduced novel estimator called Hybrid Density Filter relies on approximating the nonlinear system, thus approximating conditional densities. These densities nonlinearly relate the current system state to the future system state at predictions or to potential measurements at measurement updates. A hybrid density consisting of both Dirac delta functions and Gaussian densities is used for an optimal approximation. This paper addresses the optimization problem for treating the conditional density approximation. Furthermore, efficient estimation algorithms are derived based upon the special structure of the hybrid density, which yield a Gaussian mixture representation of the system state's density.

This paper addresses the problem of model-based reconstruction and parameter identification of distributed phenomena characterized by partial differential equations. The novelty of the proposed method is the systematic approach and the integrated treatment of uncertainties, which naturally occur in the physical system and arise from noisy measurements. The main challenge of accurate reconstruction is that model parameters, i.e., diffusion coefficients, of the physical model are not known in advance and usually need to be identified. Generally, the problem of parameter identification leads to a nonlinear estimation problem. Hence, a novel efficient recursive procedure is employed. Unlike other estimators, the so-called Hybrid Density Filter not only assures accurate estimation results for nonlinear systems, but also offers an efficient processing. By this means it is possible to reconstruct and identify distributed phenomena monitored by autonomous wireless sensor networks. The performance of the proposed estimation method is demonstrated by means of simulations.

For several tasks in sensor networks, such as localization, information fusion,or sensor scheduling, Bayesian estimation is of paramount importance. Due to the limited computational and memory resources of the nodes in a sensor network, evaluation of the prediction step of the Bayesian estimator has to be performed very effciently. An exact and closed-form representation of the predicted probability density function of the system state is typically impossible to obtain, since exactly solving the prediction step for non-linear discrete-time dynamic systems in closed form is unfeasible. Assuming additive noise, we propose an accurate approximation of the predicted density, that can be calculated effciently by optimally approximating the transition density using a hybrid density. A hybrid density consists of two different density types: Dirac delta functions that cover the domain of the current density of the system state, and another density type, e.g. Gaussian densities, that cover the domain of the predicted density. The freely selectable, second density type of the hybrid density depends strongly on the noise affecting the nonlinear system. So, the proposed approximation framework for nonlinear prediction is not restricted to a specific noise density. It further allows an analytical evaluation of the Chapman-Kolmogorov prediction equation and can be interpreted as a deterministic sampling estimation approach. In contrast to methods using random sampling like particle filters, a dramatic reduction in the number of components and a subsequent decrease in computation time for approximating the predicted density is gained.

For the collaborative control of a team of robots, a set of well-suited high-level control algorithms, especially for path planning and measurement scheduling, is essential. The quality of these control algorithms can be significantly increased by considering uncertainties that arise, e.g. from noisy measurements or system model abstraction, by incorporating stochastic filters into the control. To develop these kinds of algorithms and to prove their effectiveness, obviously real- world experiments with real world uncertainties are mandatory. Therefore, a test-environment for evaluating algorithms for collaborative control of a team of robots is presented. This test-environment is founded on miniature walking robots with six degrees of freedom. Their novel locomotion concept not only allows them to move in a wide variety of different motion patterns far beyond the possibilities of traditionally employed wheel-based robots, but also to handle real-world conditions like uneven ground or small obstacles. These robots are embedded in a modular test-environment, comprising infrastructure and simulation modules as well as a high-level control module with submodules for pose estimation, path planning, and measurement scheduling. The interaction of the individual modules of the introduced test-environment is illustrated by an experiment from the field of cooperative localization with focus on measurement scheduling, where the robots that perform distance measurements are selected based on a novel criterion, the normalized mutual Mahalanobis distance.