The physical collaboration between humans and robots in industrial applications presents a transformative approach to automation by merging human capabilities with robotic efficiency. This paper introduces an innovative active sensor system designed to ensure safe cooperation between humans and large-scale robots. The system employs camera-monitored LED flooring to create dynamic safety zones, enabling real-time monitoring and visualization of workspaces. By adhering to fail-safe principles, this technology addresses the inherent risks associated with high-speed and heavy-load robotic operations. The proposed solution not only enhances safety but also optimizes workspace utilization, fostering greater acceptance of robotic systems. This work outlines the operational principles, technological advancements, and potential applications of this ground-breaking technology.

The reliability of systems that use machine learning to recognize the human working in an industrial environment is of high importance for the employee safety. we present a framework which is capable of recognizing the person's natural interaction with an industrial machine. We focus on the application of human action recognition in the context of machine operation by skilled workers in industrial or commercial environments. We propose a framework that includes action recognition as part of a software component for understanding behavior. For our use case, we defined an exemplary machine operation workflow which we use to compare five different neural networks in terms of prediction accuracy and real-time capabilities. Moreover, we compare different input shapes as the resolution of input images and the size of the possible 3D-volume in order to study the robustness of the models. For our evaluation, we created our own custom dataset containing six action classes. Our analysis shows that the best model is the I3D with color images, a resolution of 112 × 112 pixels and 16 consecutive frames. The I3D also exhibited the best run-time performance for real-time applications.

Projective Augmented Reality (AR) systems are widely used in robotics. Main applications include information visualization to illustrate the intention of a robot or mobile platform, failure modes or teaching. A new topic in the field of projective AR for human-robot cooperation (HRC) states the visualization of safety-related information. In this industry practice paper, we present a system and method that realizes the illustration of safety zones monitored by laser-scanners. Furthermore, the system was build-up at a real industrial application with human- robot coexistence in a shared workspace.

The reliability and robustness of systems using machine learning to detect humans is of high importance for the safety of workers in a shared workspace. Developments such as deep learning are advancing rapidly, supporting the field of robotics through increased perception capabilities. An early detection of humans will support robot behavior to reduce downtime or system stoppages due to unsafe proximity between humans and robots. In this work, we present an industry-oriented experimental setup, in which humans and robots share the same workplace. We have created our own dataset to detect humans wearing different clothing. We evaluate Faster R-CNN and SSD which are state-of-the-art detectors on two different camera viewpoints. In addition, this paper elaborates on the requirements for validating the safety of such a system to be used in industrial safety applications.

The growing trend in the manufacturing industry towards human-robot collaboration as a new paradigm has the potential to increase the efficiency of robot-enabled production lines. This development requires safety-oriented sensor systems, frameworks and reliable algorithms in order to avoid hazardous situations. These systems should be able to distinguish between different objects, mobile vehicles and human workers to allow adaptive control strategies of the robots. In this paper, we introduce a tactile flooring system with high spatial resolution to determine the humans in a workplace and to continue the robot's task when mobile units are entering the workspace and no life-threatening situation is expected. The pressure data of the tactile floor is transformed to image data which is used to determine the objects presented. There are several categories to distinguish between specific vehicles, humans and users resulting in a multi-label classification problem with up to three categories presented in one pattern. Our CNN classifies five different objects with an exact match of 75%.

This paper presents the development of an apparatus for sensitive catching of guided, heavy masses (10...13 kg). The apparatus uses a spring-supported tactile sensor to measure contacts, respectively forces, and two magnetic grippers for catching and holding the drop-weight. The tactile-sensor stamp allows a dexterous catching of the drop-weight by slightly moving the gripper away to reduce harsh physical impacts. This research is based on a project to automate the calibration process for the so-called light drop-weight tester. Such drop tests are currently executed manually, as they require a high degree of dexterity. However, this work is unergonomic, with high repetitions of a load. Additionally, the operator has to perform all tests with a high degree of sensitivity, since the measuring results are adversely affected by impacts and the resulting mechanical vibrations. The single axis catching apparatus developed by the Fraunhofer IFF solves the catching problem solely through use of tactile information (e.g. a vision system is not required). The developed solutions could be modified for a similar apparatus with multiple degrees of freedom.

Brain-machine interfaces (BMI) open new opportunities to control robotic devices as they provide the feasibility to translate brain signals into commands. Severely motor impaired people who have lost muscle control could benefit from this technique to control assistive devices, which support them in daily life. However, non-invasive BMIs can distinguish only a few different commands with relatively high error rates, which makes the asynchronous control of a robot with multiple degrees of freedom challenging. Here, we introduce a novel robotic grasping system, which combines scene recognition techniques and autonomous path planning with user interaction instantiated by a hybrid control system based on the electroencephalogram and the electrooculogram. The results show that healthy subjects can reliably perform a grasp-and-place task, arranging four objects at defined positions within 133-331s (193.6 ±61.5s), while they require only a few corrections. Our robot control platform proved to work solely with electrophysiological control signals and thus, constitutes a basis to perform various robot actions initiated by motor-impaired people.

This paper presents a robot control architecture comprised of tactile and vision sensors incorporated into an off-the-shelf bin picking system. The proposed architecture facilitates flexible and reliable handling of objects and materials by employing a tactile grasp validation system and in-hand object monitoring. Two industrial grippers, specifically a magnetic gripper and a flexible vacuum gripper, are equipped with a compliant custom tactile sensor array. The algorithms used are for each specific gripper and respective sensor, however are transferrable to other grippers as well. The tactile sensing augments vision-based picking approaches, thus supporting in-hand object recognition [1] that is particularly useful for hard-to-recognize objects such as objects with transparent or shiny surfaces. Algorithms for tactile-based object pose estimation in the gripper and for grasp monitoring, including grasp validation, complete the approach.

Robot-based sensitive assembly is a recent and growing trend in robotics. Force-controlled robots are expected to interact with an unknown environment using solely force feedback information. In general, the exact contact is difficult to predict due to various impact factors, such as the dynamics of the interaction, work-piece stiffness and geometry, the robot's configuration, and the efficiency of the control algorithm. Currently, there is no general indicator for evaluating the performance of a force-controlled robot. This work presents a concept of such a performance indicator. In order to test the proposed concept for comparison, an experimental setup is presented that simulates a contour-following task under force control. This setup is used to test two robots with different force-controllers and control principles, namely direct force and impedance control. The results indicate good applicability of the proposed performance indicator to benchmark force-controlled robots, and this is extensively discussed.

In this work, preparations for the nonparametric calibration of a 7 DOF light-weight robot using machine learning techniques are presented. The approach was developed to satisfy the requirements on absolute accuracy for robot-assisted surgery. With the kinematic and non-kinematic properties in mind, we showed that a decomposition of the robot's kinematic chain can drastically reduce the number of necessary samples for a sophisticated training set. Thus, the data acquisition can be accomplished in a feasible time frame. Furthermore, we cope with the problem of data registration between the robot's internal model and the external measurements. We can show that by carefully choosing the split point for the decomposition, errors caused by the dependency between sub-chains of the robot are small enough to yield satisfying results.

During minimally-invasive interventions, physicians need to interact with medical image data, which cannot be done while the hands are occupied. To address this challenge, we propose two interaction techniques which use gaze and foot as input modalities for hands-free interaction. To investigate the feasibility of these techniques, we created a setup consisting of a mobile eye-tracking device, a tactile floor, two laptops, and the large screen of an angiography suite. We conducted a user study to evaluate how to navigate medical images without the need for hand interaction. Both multimodal approaches, as well as a foot-only interaction technique, were compared regarding task completion time and subjective workload. The results revealed comparable performance of all methods. Selection is accomplished faster via gaze than with a foot only approach, but gaze and foot easily interfere when used at the same time. This paper contributes to HCI by providing techniques and evaluation results for combined gaze and foot interaction when standing. Our method may enable more effective computer interactions in the operating room, resulting in a more beneficial use of medical information.

In this paper a method is presented that allows an operator to hand-guide a robot along a predefined geometric path. This is a common use case in robot assisted surgery, which often has high demands on precision. In order to ensure the path accuracy of the robot, joint velocity and joint acceleration constraints are enforced to prevent undesired saturation effects of the actuators. Furthermore, necessary optimization steps are calculated in an offline phase and utilized during runtime to ensure realtime capabilities. The functionality of the method is evaluated using simulated sensor readings, controlling a kinematic model of the robot. While the focus is on surgical applications, the method can be useful in other domains as well, e.g. rehabilitation robotics or industrial applications.

In this contribution we present an innovative and trendsetting solution for safeguarding human-robot cooperative workplaces with high-payload robots through a combination of safeguarding technologies addressing both hard1- and soft2- safety considerations. This consists of a tactile floor with spatial resolution as a hard- safety sensor for workspace monitoring together with a projection system as a soft- safety component to visualize the boundaries of the safety zones. This safety concept is capable of establishing both manually defined safety zones and dynamically generated safety zones that are based on the current robot's joint positions and velocities, thus offering a maximum of free space around the robot to the user. The paper aims on introducing the novel safety concept and will briefly describe the development of the underlying technologies.

Capable mobile manipulators require sophisticated sensing and environment perception. An important task is the precise localization of objects to be handled. In order to be able to deal with a wide variety of applications, sensor systems need to be able to work with glossy, shiny or transparent objects. Here we present an approach to that problem based on analyzing specular reflections using a custom build, head mounted light-field camera. We derive a suitable algorithm including highlight feature detection as well as highlight depth estimation and will present exemplary results.

In order to define meaningful limit values for human-robot collisions, we have come to the conclusion that only comprehensive collision tests with live test subjects will successfully lead to verified limit values. A literature survey about current approaches for limiting the consequences of hazardous contacts between humans and moving machines (including robots) showed that limit values are often specified without an appropriate injury severity or without considering the variety of the human body. In this article, we show why the injury onset is appropriate to limit the consequences of human-robot collisions, and how this onset can be quantified through collision tests with live test subjects. With our work we would like to present a promising method that can remedy the current lack of consensus regarding an appropriate injury severity and verified limit values.

This paper presents the application of a novel projection-based safety system for ensuring hard safety in human-robot collaboration. We adapted the proposed sensor system to incorporate the joint positions and velocities of a collaborative robot, thus offering the opportunity to establish minimal and well-shaped safety spaces around the robot at any time. In this contribution we explain in detail main challenges and their solutions for generating and monitoring such safety spaces. Furthermore, we build up a future collaborative workplace to demonstrate the practicability of the system under operational conditions.

Human-robot interaction in a shared workspace permits and often even requires physical contact between humans and robots. A key technology to ensure that physical human robot interaction is safe is to monitor contact forces by providing the robot with a tactile sensor as an artificial skin.

In this paper, the kinematics of a joint with three degrees of freedom for a novel hyper-redundant robot arm is investigated. This joint is able to achieve a superposed rotation about two axes (roll and pitch) and a translational motion along one axis as a common prismatic joint. The forward and inverse kinematics of the so-called Multi-Joint will be determined in a closed-form solution. Next, the reachable and suitable workspace will be worked out to determine the solution space of the inverse kinematics. Finally, the differential kinematics to compute the cartesian velocity and statics will be determined.

This paper presents a new approach to safety and transparency in physical human-robot collaboration by using conventional projector and camera equipment. Our system is able to establish an arbitrarily shaped light barrier around the workspace of a robot. The barrier can be made dynamic, i.e. it can adapt to the actual movement of the robot. This is a highly desirable property in scenarios requiring human robot collaboration. We will discuss a method for visually detecting violations of the safety barrier including compensation of ambient light. Furthermore, we argue that this method complies with several requirements regarding reliability in safety critical systems.

Large waste water pipes are a hazardous working environment for humans. Nevertheless such pipes must be inspected on a regular basis. Many of todaypsilas inspection systems for underground sewer pipes consist of a single TV-camera and are designed for pipes smaller than one meter in diameter. The most recent automatic inspection systems are equipped with advanced sensors that make it possible for an operator to perform this task from an outside position for larger pipes (diameter between 1.4 to 2.8 meters). The cable-guided damage surveying system, SEK, (see Fig. 1) was developed by the Fraunhofer IFF on behalf of the Emschergenossenschaft to be a versatile and easy-to-use tool to detect various kinds of damages above and below the water-line with high accuracy.

The Emschergenossenschaft based in Germany is currently planning the Emscher sewer system, arguably the largest residential water management project in Europe in years to come. The Emschergenossenschaft engaged the Fraunhofer Institute for Factory Operation and Automation (IFF) in Magdeburg, Germany, as the general contractor to develop automatic inspection and cleaning systems to meet the requirements imposed by legal guidelines. The systems must operate continuously in a sewer line that has diameters ranging from 1400 to 2800 mm and is partially filled, 25% at minimum, all the time. To construct the Emscher sewer system, the Emschergenossenschaft favors a one-pipe line in long sections. A walk-through or inspection by personnel would be impossible in these sections. The Fraunhofer Institute IFF has developed prototypes of all systems for motion through the sewer and all sensor systems, thus achieving a new quality of inspection above and below the water line under these difficult conditions. This article describes significant project results and important components of inspection such as the inspection systems, pipe axis measurement, system positioning, and sensor systems for damage detection. Fundamental for the development of the inspection systems is the detail of the inspection, which goes far beyond the video inspection common today, and the ability to take comparative measurements throughout the sewer system's period of operation in order to track the development of damage.

The Emschergenossenschaft based in Germany is currently planning the Emscher sewer system, arguably the largest residential water management project in Europe in years to come. In 2002, the Emschergenossenschaft engaged the Fraunhofer Institute for Factory Operation and Automation (IFF) in Magdeburg, Germany, as the general contractor to develop automatic inspection and cleaning systems to meet the requirements imposed by legal guidelines. The systems must operate continuously in a sewer line that has diameters ranging from 1400 to 2800 mm and is partially filled, 25% at minimum, all the time. To construct the Emscher sewer system, the Emschergenossenschaft favors a one-pipe line in long sections. A walk-through or inspection by personnel would be impossible in these sections. The Fraunhofer Institute IFF has developed prototypes of all systems for motion through the sewer and all sensor systems, thus achieving a new quality of inspection under these difficult conditions. This article describes significant project results and important components of inspection such as the inspection systems, pipe axis measurement, system positioning and sensor systems for damage detection. Fundamental for the development of the inspection systems are the detail of the inspection that far surpasses the video inspection common today and the capability to take comparative measurements throughout the sewer system's period of operation in order to track the development of damage

The Fraunhofer Institute for Factory Operation and Automation (IFF) is intensively exploring possibilities for robots to engage in various service tasks, especially fully-automatic systems for facade cleaning. We have already designed and built a variety of different facade cleaning robots and concepts. These robots and concepts are based on various motion systems (i.e. walking mechanisms, wheeled vehicles, balloon-based systems, etc.) that are specially-suited for motion along different building types. This paper gives an overview about different facade cleaning robots developed by the Fraunhofer IFF The facade cleaning robot, SIRIUSc, for use on skyscrapers, the robot to clean the 25,000 m/sup 2/ vaulted glass hall of the Leipzig Trade Fair in Germany, as well as the completed concept for a balloon-based robot for cleaning the inner side of atriums and glass roofs are discussed here. The unique aspects of the main components of these robots will be addressed in particular.