Robot programming for complex assembly tasks is challenging and demands expert knowledge. With Augmented Reality (AR), immersive 3D visualization can be placed in the robot's intrinsic coordinate system to support robot programming. However, AR interfaces introduce usability challenges. To address these, we introduce a hybrid user interface (HUI) that combines a 2D desktop, a smartphone, and an AR head-mounted display (HMD) application, enabling operators to choose the most suitable device for each sub-task. The evaluation with an expert user study shows that an HUI can enhance efficiency and user experience by selecting the appropriate device for each sub-task. Generally, the HMD is preferred for tasks involving 3D content, the desktop for creating the program structure and parametrization, and the smartphone for mobile parametrization. However, the device selection depends on individual user characteristics and their familiarity with the devices.

To this day, only a small number of industrial robots is used in assembly. One key reason for this is that specific contact situations require the introduction of force-control schemes. The parameters for those schemes are hard to select in practice, because they require in-depth expertise about the robot and the process. Learning-from-Demonstration (LfD) provides a powerful approach to intuitively parameterize robot programs by demonstrating the task at hand. However, when dimensions increase by including force or orientation, many LfD algorithms are hard to verify, understand and maintain, requiring expert knowledge to make adaptions, effectively making it a "black-box". This property renders them ineffective for usage in industrial applications. We build upon a system of composable skills, that can be easily adapted by experts without the need to demonstrate the task again. This approach to skill-based robot programming promises to address the issues of readability and maintainability by sequencing robot movements in skills and breaking them down into understandable (sub-)goals. In this paper, we combine skill-based programming with LfD, preserving both maintainability and intuitive parameterization. We present (a) an approach to parameterize and create sequences of hierarchies of force- and/or position-controlled robot skills from a LfD model, (b) which can be adapted by a user by hand with few, basic and understandable parameters, and (c) show its applicability on the real-world example of terminal clamp assembly. We achieve a reduction in teach-in time of 53.8% for variants, increased robustness against variance, and efficient tight stacking of clamps with a gap of ≤ 1mm.

Taking over arbitrary tasks like humans do with a mobile service robot in open-world settings requires a holistic scene perception for decision-making and high-level control. This article presents a human-inspired scene perception model to minimize the gap between human and robotic capabilities. The approach takes over fundamental neuroscience concepts, such as a triplet perception split into recognition, knowledge representation, and knowledge interpretation. A recognition system splits the background and foreground to integrate exchangeable image-based object detectors and simultaneous localization and mapping, a multilayer knowledge base represents scene information in a hierarchical structure and offers interfaces for high-level control, and knowledge interpretation methods deploy spatio-temporal scene analysis and perceptual learning for self-adjustment. A single-setting ablation study is used to evaluate the impact of each component on the overall performance for a fetch-and-carry scenario in two simulated and one real-world environment.

Skill-based programming has proven to be advantageous for assembly tasks, but still requires expert knowledge, especially for force-controlled applications. However, it is error-prone due to the multitude of parameters, e.g. different coordinate frames and either position-, velocity- or force-controlled motions on the axes of a frame. We propose a mixed reality based solution, which systematically visualizes the geometric constraints of advanced high-level skills directly in the real-world robotic environment and provides a user interface to create applications efficiently and safely in mixed reality. Therefore, state-machine information is also visualized, and a holographic digital twin allows the user to ad-hoc validate the program via force-enabled simulation. The approach is evaluated on a top hat rail mounting task, proving the capability of the system to handle advanced assembly programming tasks efficiently and tangibly.

Robotic applications like automated order picking in warehouses or retail stores, or fetch and carry tasks in hospitals, care homes, or households rely on the capability of service robots to find and handle a specific type of object. These applications are challenging as the set of objects is very large and varies over time. Despite its significance, there is no suitable universal large-scale dataset available from the retail domain, which allows for a principled analysis of all relevant robotics research aspects in that field. Hence, this paper introduces a novel dataset of more than 1,000 retail objects, including color images, 3D scans, and high-resolution textured 3D models of individual objects, synthetic scenes and real settings, which covers the specifics of the retail domain. The dataset was designed to serve researchers in all relevant robotics tasks in retail like 3D reconstruction and object modeling, large-scale object classification and instance detection including incremental learning and fine-grained detection, text reading, logo detection, semantic grounding and affordance detection, grasp analysis and manipulation planning, as well as digital twinning and virtual environments. Based on synthetic RGB images of scenes created from the 3D models, two exemplary use cases are examined in this paper to demonstrate the benefits of the dataset: we evaluate the state-of-the-art incremental object detection method InstanceNet and a few-shot fine-grained object classification method. The results prove the suitability of InstanceNet for incremental object detection on large datasets and are promising for the few-shot object classification system.

For various automated palletizing tasks, the detection of packaging units is a crucial step preceding the actual handling of the packaging units by an industrial robot. We propose an approach to this challenging problem that is fully trained on synthetically generated data and can be robustly applied to arbitrary real world packaging units without further training or setup effort. The proposed approach is able to handle sparse and low quality sensor data, can exploit prior knowledge if available and generalizes well to a wide range of products and application scenarios. To demonstrate the practical use of our approach, we conduct an extensive evaluation on real-world data with a wide range of different retail products. Further, we integrated our approach in a lab demonstrator and a commercial solution will be marketed through an industrial partner.

In manufacturing, the current trend-shift from mass-production to mass-personalization is enabled, among others, by the emerging field of human-robot collaboration (HRC), in which humans collaborate or work in proximity with robots. In HRC scenarios, robots need to exert a desired behaviour that maximizes utility without sacrificing safety and responsiveness. To maximize safety and utility in static environments, state-of-the-art offline motion-planners use computationally-heavy algorithms for approximating the collision-free robot reachability and accordingly generate (sub-)optimal robot trajectories. To enable real-time responsiveness, we propose an integrated global planner to generate sub-optimal trajectories. It relies on a closed-loop reactive controller for executing the global plan while ensuring safety with practical assumptions about the environment. We evaluate GLIR in simulation. In our experiments, our global planner operates at 25 Hz and the local planner at 100 Hz, enabling their execution in dynamic environments. In all experiments on static scenes with static and dynamic goals, GLIR keeps a safety distance from obstacles. We showcase some simulation experiments and a real-world demonstration in the video available at https://mohamedgalil.github.io/glir/.

When robots are used to automate repetitive production tasks, the productivity of the manufacturing system crucially depends on the robot's task execution speed. An out-of-the-box solution is typically slow, whereas achieving shorter cycle times typically requires large efforts with respect to controller design and tuning. This dilemma can be resolved by learning control algorithms that autonomously improve performance without requiring any system-specific tuning. In the present work, we propose a novel learning control scheme that autonomously reduces the execution times of robotic systems that perform repetitive manufacturing tasks. To this end, we combine an Iterative Learning Control (ILC) approach with a trial-varying reference adaptation. The reference trajectory is slowly adapted to ensure that the given task is performed successfully on every single iteration without constraint violations. Therefore, the learning process can be carried out during operation. We validate the practical applicability of the method by real-world experiments on a 6-axis robot that performs a linear motion and a contact-force task. Despite the fundamentally different characteristics of these two tasks, the proposed algorithm achieves a remarkable reduction of cycle times, namely, by a factor of 4 in the linear motion task and a factor of 10 in the contact-force task. These results provide an important step toward robotic manufacturing systems that autonomously optimize their own performance during operation.

Increasingly volatile markets challenge companies and demand flexible production systems that can be quickly adapted to new conditions. Machine Learning has proven to show significant potential in supporting the human operator during the time-consuming and complex task of robot pro-gramming by identifying relevant parameters of the underlying robot control program. We present a solution to learn these parameters for contact-rich, force-controlled assembly tasks from a simulation using hardware-independent robot skills. We show that successful learning and real-world execution are possible even under process deviation and tolerances utilizing the designed learning system. We present learning skill param-eters as high-level robot control, evaluation and comparison of extensive simulations, and preliminary experiments on a physical robot test-bed. The developed solution approach is evaluated and discussed using the Peg-in-Hole process, a typical benchmark process in force-controlled assembly.

In this paper, we present a Domain Randomization and a Domain Adaptation approach to transfer experience for entanglement detection and separation from simulation into a real-world bin-picking application. We investigate the influence of different randomization options in image processing and use a CycleGAN as a further Domain Adaptation method to synthesize simulation data as realistically as possible. On the basis of this adapted data we re-train our detection and separation methods and validate the usefulness of these Sim-to-Real methods. In numerous real-world experiments we show that we achieve a significant increase of up to 71.74 % in the performance of the overall system by using the Sim-to-Real approaches as opposed to the direct transfer.

This paper introduces a novel approach for the grasping and precise placement of various known rigid objects using multiple grippers within highly cluttered scenes. Using a single depth image of the scene, our method estimates multiple 6D object poses together with an object class, a pose distance for object pose estimation, and a pose distance from a target pose for object placement for each automatically obtained grasp pose with a single forward pass of a neural network.By incorporating model knowledge into the system, our approach has higher success rates for grasping than state-of-the-art model-free approaches. Furthermore, our method chooses grasps that result in significantly more precise object placements than prior model-based work.

Programming-by-Demonstration (PbD) or Imitation Learning (IL) provides a powerful approach to program robots intuitively. In industrial settings, these approaches are not commonly deployed since several factors negatively impact their productive use. The most common reasons are safety concerns on various levels. First, industrial robots are only allowed to be operated in direct contact if the operator has sufficient experience. Secondly, many PbD systems do not incorporate force measurements in their model directly. This renders them ineffective for assembly tasks, such as snap- fit connections. In this paper we (a) present an approach to incorporate force measurements into a generative model (b) using only external sensors without relying on measurements from the robot during demonstrating to decouple the teaching process from the robot and (c) show the benefit of explicit force measurement and modeling on a Franka Emika Panda robot by the example of assembling terminal clamps on a DIN rail.

Cable-driven parallel robots, hereinafter referred to as cable robots, use cables to manipulate a mobile platform with 6 DOF. Cable robots have a low moved mass, as the winches with the servo drives are fixed to the machine frame and light weight synthetic fiber cables can be used. Therefore, cable robots are assumed to have a good energy efficiency. To analyze the energy efficiency in detail, we establish an energy consumption model for a cable robot and parametrize it for the cable robot IPAnema 3. Losses in the mechanical parts like winches as well as electrical losses in the servo amplifier and recuperation effects are taken into account. The analysis of the energy consumption shows that the mechanical losses are dominant. The losses occur especially during movement of the robot while the energy needed for statically balancing the load is quite low. We can experimentally determine a maximum winch efficiency of 85%. For dimensioning the drive, one has to add approximately one third of the torque to account for the friction in the mechanics. In a fully-constrained cable robot, the energy consumption can be influenced by the internal tension in a range of 20%. We also compare the energy efficiency of the cable robot with an industrial robot. The comparison shows, that both robots consume almost the same amount of energy.

Workspace determination for robots is an important step in analysis and synthesis. A couple of methods for computing the wrench-closure workspace of cable-driven parallel robots were reported in the literature but all methods tend to be time consuming. In this paper, a new algorithm is presented that exploits different techniques to speed up the computation. Pre-computation is largely exploited and benefit are gained both from considerations in computer algebra and efficient numerical routines. Results from the computation of the translational (sometimes also called constant orientation) wrench-closure workspace are presented and performance values are provided. To the best of the authors' knowledge, the method proposed in this paper is superior in terms of computational time to any other approach for workspace computation.

In a cable-driven parallel robot, elastic cables are used to manipulate the end effector within the workspace. Cable force measurement is necessary for several control algorithms like cable force control, contact control, or load identification. The cable force sensor can be placed directly at the connection point on the platform or somewhere along the cable using pulleys. The pulleys between the force sensor and the platform disturb the force measurement accuracy due to friction. This paper deals with modeling and compensation of the friction. The friction behavior in the drive train with focus on the effects of the pulleys is non-trivial, as the cable movement consists of microscopic and macroscopic movements and standstills. Friction models from Coulomb and Dahl are adapted to deal with the pulley friction. The experimental evaluation showed an improvement of 70% with respect to the uncompensated case.

In a cable-driven parallel robot, elastic cables are used to manipulate the end effector in the workspace. In this paper we present a dynamic analysis and system identification for the complete actuator unit of a cable robot including servo controller, winch, cable, cable force sensor and field bus communication. We establish a second-order system with dead time as an analagous model. Based on this investigation, we propose the design and stability analysis of a cable force controller. We present the implementation of feed-forward and integral controllers based on a stiffness model of the cables. As the platform position is not observable the challenge is to control the cable force while maintaining the positional accuracy. Experimental evaluation of the force controller shows, that the absolute positional accuracy is even improved.

In Cable-Driven parallel robots, elastic cables are used to control the movement of a platform in the workspace. The accuracy of a robot system is one of the most important comparison criteria for various applications. In this paper, we present a new approach to increase the relative accuracy of a cable robot under changing payload, by compensating the error induced by elasticity. In the first step, the load is identified based on cable force sensors and the acceleration of the platform. Second, for the actual pose the stiffness matrix is computed and the expected displacement of the platform in Cartesian space is determined. Lastly, the displacement is compensated by adapting the cable lengths. Furthermore, the influence of the calibration of cable force sensors and their parameters' inaccuracies are discussed. The algorithms are implemented and evaluated on a 6-DOF cable robot. We could experimentally prove an improvement of over 50% in the relative accuracy under changing load.