With a growing number of robots being deployed across diverse applications, robust multimodal anomaly detection becomes increasingly important. In robotic manipulation, failures typically arise from (1) robot-driven anomalies due to an insufficient task model or hardware limitations, and (2) environment-driven anomalies caused by dynamic environmental changes or external interferences. Conventional anomaly detection methods focus either on the first by low-level statistical modeling of proprioceptive signals or the second by deep learning-based visual environment observation, each with different computational and training data requirements. To effectively capture anomalies from both sources, we propose a mixture-of-experts framework that integrates the complementary detection mechanisms with a visual-language model for environment monitoring and a Gaussian-mixture regression-based detector for tracking deviations in interaction forces and robot motions. We introduce a confidence-based fusion mechanism that dynamically selects the most reliable detector for each situation. We evaluate our approach on both household and industrial tasks using two robotic systems, demonstrating a 60% reduction in detection delay while improving frame-wise anomaly detection performance compared to individual detectors.

This work aims at facilitating the integration of industrial robots and other devices such as their gripper tools at small and medium-sized enterprises (SMEs). For this purpose, an intuitive user interface for the skill-based instruction of robot systems is combined with standardized opc UA-based skill interfaces that support various hardware and software resources from different manufacturers. Special emphasis is laid on supporting different user groups with varying levels of expertise. Production system engineers are provided with a detailed graphical user interface (GUI) for hierarchically defining new skills by combining preexisting ones. System operators receive a simplified view with limited complexity for process instruction and changing high-level task parameterizations. The skills and relevant semantic context knowledge about products, processes, and resources (PPR) are formally represented in OWL ontologies to enable hardware-agnostic process descriptions that can be deployed to different production environments, while automatically deriving parameterizations for skill invocations. The proposed concept has been qualitatively evaluated in two real-world robot workcells based on a smartphone accessory packaging use case.

To perform tasks in unstructured environments, robots need to be able to apply learned skills to different contexts and to autonomously make decisions online. We, therefore, developed a novel data-driven task learning approach that segments a task demonstration into simpler skills and structures them in a high-level task graph. In contrast to other state-of-the-art methods, the presented approach can not only infer the low-level skills and their respective subgoals but also multimodal feature constraints fitted individually to each skill. The inferred feature constraints allow to detect anomalies during autonomous task execution, which can be automatically resolved by a recovery behavior of the task graph. The subgoals encode each skill's intention and thereby enable to flexibly transition between skills and to generalize the behavior to new setups. By separating the subgoal and constraint inference, we achieve a reduced computational complexity and an increased performance compared to state-of-the-art task learning approaches. In a real-world manipulation task, we demonstrate the reusability of skills as well as the autonomous decision-making of our approach.

Conventional robot programming methods are not suited for non-experts to intuitively teach robots new tasks. For this reason, the potential of collaborative robots for production cannot yet be fully exploited. In this work, we propose an active learning framework, in which the robot and the user collaborate to incrementally program a complex task. Starting with a basic model, the robot's task knowledge can be extended over time if new situations require additional skills. An on-line anomaly detection algorithm therefore automatically identifies new situations during task execution by monitoring the deviation between measured- and commanded sensor values. The robot then triggers a teaching phase, in which the user decides to either refine an existing skill or demonstrate a new skill. The different skills of a task are encoded in separate probabilistic models and structured in a high-level graph, guaranteeing robust execution and successful transition between skills. In the experiments, our approach is compared to two state-of-the-art Programming by Demonstration frameworks on a real system. Increased intuitiveness and task performance of the method can be shown, allowing shop-floor workers to program industrial tasks with our framework.