Human-robot collaboration in shared workspaces enables companies to improve efficiency and the quality of work for human workers. A novel research direction in this field is that human and robot dynamically negotiate which actions to perform. This requires the robot to permanently monitor the current task state and actions the human has performed. We envision a system that tracks the task progress based on visual input. We assume that a task specification is provided in terms of a coloured Petri net. The contribution of this paper is twofold. We introduce the notion of emissions for coloured Petri nets that incorporate partial observability and uncertainty. We then show how one can efficiently determine the evolution of the net when a sequence of partial observations is provided. To this end, we determine a small set of transition as candidates in a firing sequence first. Then, transition sequences are sampled to obtain markings compatible with the latest observation. We evaluate our algorithm in a simulated environment on a pick and place task and compare it to an aging-based approach from literature. Results show that our algorithm achieves higher precision compared to the aging-based approach. Running times indicate that the update procedure can run at 15 Hz on average.

Robust and reliable perception plays an important role when humans engage into cooperation with robots in industrial or household settings. Various explicit and implicit communication modalities and perception methods can be used to recognize expressed intentions. Depending on the modality, different sensors, areas of observation, and perception methods need to be utilized. More modalities increase the complexity and costs of the setup. We consider the scenario of a cooperative task in a potentially noisy environment, where verbal communication is hardly feasible. Our goal is to investigate the importance of different, non-verbal communication modalities for intention recognition. To this end, we build upon an established benchmark study for human cooperation and investigate which input modalities contribute most towards recognizing the expressed intention. To measure the detection rate, we conducted a second study. Participants had to predict actions based on a stream of symbolic input data. Findings confirm the existence of a common gesture dictionary and the importance of hand tracking for action prediction when the number of feasible actions increases. The contribution of this work is a usage ranking of gestures and a comparison of input modalities to improve prediction capabilities in human-robot cooperation.

We propose a highly flexible approach to human-robot cooperation, where a robot dynamically selects operations contributing to a shared goal from a given task model. Therefore, knowledge on the task progress is extracted from a world model constructed from eye-in-hand camera images. Data generated from such partial workspace observations is not reliable over time, as humans may interact with resources. We therefore use a human-aware world model maintaining a measure for trust in stored objects regarding recent human presence and previous task progress. Our contribution is an action selection algorithm that uses this trust measure to interleave task operations with active vision to refresh the world model. Large-scale experiments cover various sorts of human participation in different benchmark tasks through simulation of simplified, partially randomized human models. Results illuminate system behaviour and performance for different parametrizations of our human-robot teaming framework.

Flexible human-robot cooperation can be coordinated by dynamic task allocation, where team members observe task progress and repeatedly decide to execute outstanding operations from a commonly known task model. Such task-level coordination strategies do not cover cases, where a high level of abstraction for task modelling implies operations infeasible for one single agent due to limited capabilities. Task-level coordination can handle these operations by introducing additional operation-level coordination for more fine-grained interaction, e.g. to hand over a tool or provide a helping hand. We contribute an approach to classify operations regarding the interaction they require. Based on this classification, we show how robots can plan and issue communication to coordinate respective interaction within dynamic plan execution. Against the background of Smart Factories, a smartphone app is used as communication channel and evaluated in a user study.

A popular approach to intuitive robot programming is kinesthetic guiding, where the user leads the robot through a task providing a trajectory for automatized replication. While the flow of information from the user towards the robot is already well explored, the feedback from the robot to the user within the same modality is often neglected. In this work we present an innovative concept, how the robot can state an imminent collision based on an adaptive virtual kinetic friction force. Our approach features both obstacle indication and collision avoidance without additional output devices. A user study motivates certain requirements for the provided feedback, whose fulfillment is demonstrated by numerical experiments.

With a robust one-shot programming approach from previous work we are able to transfer the advantages of general purpose robots into more unstructured environments like private households or small and medium sized enterprises. From this we enable non-experts to use the capabilities of a kinesthetically guidable manipulator. The main contribution of this work is to extend our behavior-based approach by two generalizations in the task representation model, which were identified in previous work, but not yet examined. With this we introduce novel extensions of our programming paradigm by integrating task generalization in terms of path adaption and object generalization. The proposed extensions allow to generalize task solutions from a demonstration by monitoring resources (e.g. objects in the workspace) and their changes. Integrating task generalization by adapting a demonstrated task to new paths as well as new objects in similar situations raises challenges for both, demonstration and execution phase: During task demonstration it is necessary to identify which aspects can be generalized and during execution we have to distinguish adaptable from faulty situations.

Speech-based robot instruction is a promising field in private households and in small and medium-sized enterprises. It facilitates the use of robot systems for experts as well as non-experts, even while the user executes other tasks. The common approach is to map action verbs to robot tasks represented by well-defined interfaces. While this method works well for a wide variety of robot motions, it contains the necessity for defining additional robot tasks for each change in the physical properties of the manipulated objects. To overcome this drawback, we contribute a novel approach for defining complex and sensor-based robot tasks called Combined Verbalized Effects. It allows to systematically ground action verbs to sensor-based robot motions, while considering accessible tools and highlighting the motion primitive combinations complex motions are made of. Furthermore, real-world examples and a user study show the applicability of our approach.

Three dimensional models are represented in various ways. One possibility are boundary representation (B-Rep) models, which contain geometric and topological information. This makes B-Reps suitable for tasks that need an explicit algebraic representation of the surface, e.g. numerical optimization or simulation. Reconstructing a B-Rep model of an object or the environment often is a tedious task needing much manual intervention. An intuitive way is to use a hand-held depth camera and perform a real-time reconstruction. In the domain of robotics, we previously presented a system [18] that is able to incrementally reconstruct a planar B-Rep model from a stream of organized point clouds acquired from a robot mounted camera. Since the acquisition poses are known, this approach can not directly be used in a setup with a hand-held camera.

One challenge to provide the manifold benefits of a robotic system to domains like private households or small enterprises is to give non-experts an easy-to-use interface to program such systems. Although there are many intuitive robot programming paradigms, there is no general set of evaluation tools to compare them in detail. In this paper we present a toolkit to evaluate robot programming concepts. This toolkit concerns in particular the intuitiveness of a programming interface and its robustness regarding the results from generated programs. We will explain which questionnaires are necessary to analyze both aspects and suggest how to design a user study. Furthermore we provide two case examples to demonstrate the analysis and interpretation of data. As such we hope to engage robot programming developers in evaluating the acceptability of their approach, find strengths and weaknesses and finally improve the design of robots.

Robot Programming by Demonstration (PbD) allows the use of a general purpose robot without specific programming skills. The most approaches in this field use Machine Learning methods to learn a robot behavior or skill. Besides, there are a few One-Shot PbD-approaches, which reduce the programming effort to a single demonstration and therefore allow a non-expert the quick and intuitive programming of a robot. However, a common drawback of One-Shot approaches is their decreased robustness during the program execution compared to Multi-Shot approaches. This leads to our main motivation: Making One-Shot PbD programs more robust. An important aspect of robustness is the ability to detect and handle unexpected events, e.g. a dropped object or a storage place, which should be free, but is already occupied. We call such events deviations. In our previous work we proposed an approach that provides the detection and isolation of deviations by a monitoring concept using entity-based resources. The advantages of this approach include a unified detection and complete classification principle. In this paper we evaluate our concept in a user study and give several insights to learned lessons to improve usability and comparability of our approach.

We envision a system that treats humans and robots as equal partners in achieving a common goal. Equality of agents offers high flexibility through dynamic task allocation and flexible team composition, but thereby complicates the coordination of robots with human actions. We contribute a formal framework and system architecture that integrates planning of perception operations into dynamic plan execution to recognize whether subtasks have already been carried out by another agent or are ready to be executed by the robot. This is achieved by locally observing the world state with robot-mounted sensors to evaluate pre- and postconditions linked to every operation of the task. Our approach is prepared to handle arbitrary teams of multiple humans and robots. Experiments show, that our prototype is able to coordinate a stationary robot arm with human actions in pick-and-place tasks formulated as precedence graphs.

General purpose robots are established tools in a variety of industrial applications. Current household robots, however, are limited in capabilities and purpose. One reason for this is the programming effort required to utilize a general purpose robot. A popular strategy to reduce programming effort is Programming by Demonstration (PbD). Usually PbD requires several tedious demonstrations to teach a robot some behavior or skill. Notably, there are just few approaches where a single demonstration suffices. Such One-Shot approaches reduce programming effort even further and allow non-experts to quickly and intuitively program a general purpose robot. In this work we evaluate intuitiveness and robustness (i.e. the dependability when executing non-expert generated programs) of a One-Shot PbD system. To this end, we carried out a user study with 34 participants. Our participants used kinesthetic programming to complete various tasks with a light-weight robot. We instructed users with standard methods, either by speech, by graphical tutorial or by video. During the study, structured questionnaires and an observation sheet collected user impressions: The effectiveness when solving a task, the efficiency and (mental) effort during programming, the attitude towards the robot, and finally the satisfaction with the entire system. User ratings on all aspects confirm the high intuitiveness of the investigated system. In contrast to other approaches, the study did not depend on a specific instructional material to achieve high intuitiveness ratings. Overall, our findings support the advantages of a One-Shot programming system and give important insights into general requirements for robot programming by non-experts. We conclude that the investigated One-Shot PbD approach forms a vital tool for the use of robots in private households.

One long term goal of robotic research is the development of robot systems, which have approximately the same cognitive, communicational, and manipulative abilities like humans and which can be operated by non-experts. The usage by non-experts sets additional requirements to the abilities of a robot system. For instance, the system should be able to understand natural language as one of the primary communication forms. This requires knowledge about actions, which shall be executed, or about objects, which shall be manipulated. In this work, we focus on grounding knowledge for actions within the domain of object manipulation. Our approach is based on the concept of verbalized physical effects and manipulation primitives. Therefore, we introduce the concept of verbalized physical effects, describe the relations between verbal instructions, verbalized physical effects and sensor based robot motions, and evaluate our approach using a robot manipulator.

A Personal Robot should be able to handle unknown objects in unknown environments. For a manipulation task the question what to do with an object once it had been grasped is one of the most essential ones beside the grasping task itself. Moreover, the planning time should be at least as fast as the time the robot needs for its motions. We propose a fast placement planner for sensor-modelled objects in complex environments. The planner computes a stable position and orientation for the object in the environment. The algorithm uses only geometric information, most notably no force or torque sensor is required. In particular, we introduce a novel approach regarding the pose computation. By means of experiments with various household objects the robustness and performance are validated. Further on, we compare our approach with a pose computation using a physics simulation framework.

A long-term goal in current robotic research is the development of intuitive interfaces for human-robot interaction. Here, one field of application are object manipulation tasks. Such tasks consist of grasping, moving, and placing objects [1]. In this work, we focus on the subtask of moving an object, which is also called the handling of the object. We present a method for the intuitive instruction of handling tasks through verbal commands and the execution based on verbalized physical effects. We define a set of principal physical effects and describe how a physical effect can be verbalized. Furthermore, we indicate how verbal parameters can qualitatively be transformed into robot control parameters using physics. At last, we show in a user study, that the proposed method is feasible for the intuitive instruction of handling tasks to a robot system.

Programming robots is a complicated and time-consuming task. A robot is essentially a real-time, distributed embedded system. Often, control and communication paths within the system are tightly coupled to the actual physical configuration of the robot. Thus, programming a robot can hardly be achieved by experts of the domain in which the robot is used. In this paper we present an approach towards a domain specific language, which is intended to empower domain experts or even end users to specify robot programs with a dedicated background in programming techniques, but not with special knowledge in the robotics domain. Furthermore we introduce an idea to integrate a software product-line for a hardware and software transparent plug and play mechanism.

Behavior-based robotic manipulators are very flexible since they can perform many different tasks without reprogramming. Unfortunately none of the existing approaches is able to interweave multiple manipulation tasks and execute them reliably at the same time, enabeling e.g. an intuitive human-robot cooperation. To bridge this gap, we suggest a novel reactive behavior-based architecture. For this, we transfer the multitasking concept from modern computer operating systems to the robot and use a resource based approach to coordination and synchronization. With this, we are able to safely run multiple competing behaviors on a robot. Due to the design of the behaviors, new behaviors can easily be added to the system. The behaviors can be interrupted by other behaviors in order to react properly to a dynamic environment. Later, the interrupted behaviors are resumed in such a way that the system keeps a consistent state.

For effective collaboration between two humans, they are required to adapt to each other according to their apparent behavior in time. Similarly for intuitive human-robot collaboration the robot is required to adapt to the human intention apparent from his behavior. In this paper, we introduce a probability-based approach that helps the robot to adapt to the human behavior and to act proactively in the ambiguous human intentions scenarios. The robot can either wait for disambiguation of the intention, requiring extra human actions or it can proactively act depending on his previous knowledge about the human behavior. The adaptive proactivity is achieved by the help of transition weights for transitions between the neighbouring states in a finite state machine. A finite state machine models the human intention. We also introduce the online update of the intention triggers for the intentions. The online intention trigger update corresponds to the online selection of the specific state of a finite state machine as the end state. The online update of the intention triggers makes the robot more proactive in the direction of premature intention estimation. Thus the transition weights and premature intention estimation makes the human-robot collaboration more intuitive.

This paper proposes a new concept to augment direct robot programming with sensor-based branching and looping. The objective is to derive a new robot programming paradigm in order to enable non-programmers to use robots for different tasks. Inspired by the ancient programming style based on the GOTO-statement, this approach requires the user to deal with only two basic concepts: Intuitive Control Expressions and Spatial Labels. This simplicity yields a very intuitive robot programming interface. We introduce these new concepts and derive some statements about the power and the limitations of this approach.

We present a new approach that realizes an image-based fault tolerant distance computation for a multi-view camera system which conservatively approximates the shortest distance between unknown objects and 3D volumes. Our method addresses the industrial application of vision-based protective devices which are used to detect intrusions of humans into areas of dangerous machinery, in order to prevent injuries. This requires hardware redundancy for compensation of hardware failures without loss of functionality and safety. By taking sensor failures during the fusion process of distances from different cameras into account, this is realized implicitly, with the benefit of no additional hardware cost. In particular we employ multiple camera perspectives for safe and non-conservative occlusion handling of obstacles and formulate general system assumptions which are also appropriate for other applications like multi-view reconstruction methods.

The aim of this paper is to enable a programmer to easily employ external sensors for flexible robot manipulation. We describe a general approach to determine the relation between the position deviation of an object and the resulting data from sensors used to recognize this deviation. This information can be used to employ adaptation techniques to compensate the deviation thus enabling robots to react flexibly to changes such as workspace variations or object drifts. The proposed methods are designed to be independent of the type of sensor. We describe methods to automatically determine a function describing this relation and how adaptive techniques can be integrated easily into robot programs without detailed knowledge provided by the programmer.

Surgical procedures with navigation or robot system support usually require pre-operative planning data. This data can be acquired with imaging techniques such as computed tomography (CT), the current gold standard due to its high precision. With such imaging data, access trajectories, implant positions, individual milling paths, etc. can be computed. We present a novel ultrasound-based method to generate 3D image data which is well-suited for many interventions, but less costly than equivalent CT data. The method's feasibility is demonstrated for robot-based implant bed milling in the lateral skull base, in a process consisting of infrared navigation registration, manual ultrasound scan path delineation, path smoothing and checking, and robot-based ultrasound scan execution.

We present a framework for an adaptive system capable of automatically optimizing a given sensor-based program for an industrial robot to facilitate the development of fast and robust applications. We present a systematic overview of the layers of the robot controller in which a speed-up may be gained by incorporating adaptation techniques. We then measure the potential for optimization at a specific level to determine the improvement yielded with this optimization.

We present a path planner capable of efficient and real-time handling of known and unknown obstacles in highly dynamic workspaces. Known obstacles are acquired offline and stored in a world model, unknown obstacles are acquired online by one or multiple sensors. This is a typical situation for many applications. The method presented here exploits this distinction by building a static roadmap based on known obstacle information. This enables efficient path planning and real-time performance using bounded lazy evaluation thus reducing the number of costly collision test. The dynamics of the workspace are addressed by invalidation/revalidation of roadmap edges based on sensoric input. Several revalidation strategies are evaluated. The proposed path planner is probabilistically complete and utilizes global environment information to assure goal arrival, if the goal is reachable. Our approach is realized using standard PC hardware with computational requirements allowing real-time performance. Experimental results show the validity of our approach.

We present a method for quickly determining the minimum distance between multiple known and multiple unkown objects within a camera image. Known objects are objects with known geometry, position, orientation, and configuration. Unkown objects are objects which have to be detected by a vision sensor but with unkown geometry, position, orientation and configuration. The known objects are modeled and expanded in 3D and then projected into a camera image. The camera image is classified into object areas including known and unknown objects and into non-object areas. The distance is conservatively estimated by searching for the largest expansion radius where the projected model does not intersect the object areas classified as unknown in the camera image. The method requires only minimal computation times and can be used for surveillance and safety applications.

Currently industrial robotic systems employ almost exclusively global maps for navigation purposes, if any. Additional information - intra-process, spatial, current, and persistent sensor data - is useful to cope with uncertainty, measurement errors, and incompleteness of data. We propose to augment robot world models by using local sensors (which provide data from a local epsiv-environment) and build precise maps from local sensors, with force and audio classification in orthopedics applications with a medical robot system (RONAF) as an example. Improving precision of this map-building is presented both for data localisation and data insertion.

An intuitive behavior model for safe human/robot coexistence and cooperation is presented. For this purpose, we classify the robot behavior into four cooperation states, which are characterized by two criteria: motion context and interaction context. The motion context can be gross or fine and the interaction context can be free or guided. We describe basic principles relevant to modeling safe and intuitive behavior for robots. The desired robot behavior is detailed for each of the four states along with the necessary transitions between those states

In manipulation, workpieces have to be placed in an exact relationship to other objects that usually constrain the possible motions of the workpiece. Thus, control of a manipulation task can often be done by detecting the changes in the set of constraints acting on the workpiece. In the case of hard positional constraints on rigid objects, the detection can be done using simple thresholds on measured forces and/or moments. When some of these constraints are non-rigid (i. e. elastic), forces and moments vary during motions even if there is no change in the constraints, making it difficult if not impossible to find appropriate thresholds for forces and moments. In such situations changes in the constraints mostly lead to discontinuities (L e. jumps and corners) in the measured force and moment signals. This paper presents a constant time algorithm for jump and corner detection in 1-dimensional sensor signals

Current approaches to establish or improve precision of registration and execution in (surgical) robotics, especially milling applications, mostly aim at determining an exact relationship between intervention area and the robot. However, this is subject to quality deterioration due to positional shifts and deformations under load. Therefore, in this work we describe two approaches to define and measure the actual tool position more accurately - by optical and force measurements - to both ease static and dynamic registration and estimate tool and other deformations

We present an anytime fast deterministic greedy method for smoothing piecewise linear paths consisting of connected linear segments by incrementally removing path points. First, points with only a small influence on path geometry are removed, i.e. aligned or nearly aligned points. Due to the removal of less important path points, the high computational and storage space requirements of the paths are reduced and traversal of the path is accelerated. Our algorithm can be used in the most diverse applications, i.e. sweeping, path finding or programming-by-demonstration in a virtual environment

This paper discusses the handling of deformable linear objects (DLOs), such as hoses, wires, or leaf springs in a polyhedral environment. It investigates the formulation of assembly or disassembly tasks based on contact states. The result is an approach that facilitates the automatic extraction of robot programs from demonstrations in virtual reality and provides a base for the parameterization of detection algorithms. For this purpose, a contact state model for the description of assembly or disassembly tasks of DLOs is presented. It is described how the contact states can be derived from a geometric model of both the DLO and the environment. Such a model may be obtained by a simulation of the manipulation tasks in virtual reality. Further, the possible transitions between the contact states are classified into general transition classes. Those transition classes enable the selection of algorithms to detect such contact state transitions.

Currently, robotic systems employ almost exclusively global sensor information for navigation purposes. While a global map facilitates planning, it may have insufficient quality. Especially with autonomous robots, additional information - intra-process, spatial, current, and persistent sensor data - proves necessary to cope with uncertainty, measurement errors, and incompleteness of data. We aim at augmenting the world model for standard robotic systems by implementing local navigation and map-building based on local sensors (which provide local data without primary location information), with the use of force and audio classification in a medical robot system (RONAF) in orthopedics applications as an example.

It is difficult for robots to handle a vibrating deformable object. Even for human beings it is a high-risk operation to, for example, insert a vibrating linear object into a small hole. However, fast manipulation using a robot arm is not just a dream; it may be achieved if some important features of the vibration are detected online. We present an approach for fast manipulation using a force/torque sensor mounted on the robot's wrist. A template matching method is employed to recognize the vibrational phase of the deformable objects. Therefore, a fast manipulation can be performed with a high success rate, even if there is acute vibration. Experiments inserting a deformable object into a hole are conducted to test the presented method. Results demonstrate that the presented sensor-based online fast manipulation is feasible.

This paper analyzes the problem of sensor-based collision detection for an industrial robotic manipulator. A method to perform collision tests based on images taken from several stationary cameras in the work cell is presented. The collision test works entirely based on the images, and does not construct a representation of the Cartesian space. It is shown how to perform a collision test for all possible robot configurations using only a single set of images taken simultaneously.

While handling deformable linear objects (DLOs), such as hoses, wires or leaf springs, with an industrial robot at high speed, unintended and undesired oscillations that delay further operations may occur. This paper analyzes oscillations based on a simple model with one degree of freedom (DOF) and presents a method for active open-loop damping. Different ways to interpret an oscillating DLO as a system with 1 DOF lead to translational and rotational adjustment motions. Both were implemented as a manipulation skill with a separate program that can be executed immediately after any robot motion. We showed how these manipulation skills can generate the needed adjustment motions automatically based on the readings of a wrist-mounted force/torque sensor. Experiments demonstrated the effectiveness under various conditions.

Deals with the robust manipulation of deformable linear objects such as hoses or wires. We propose manipulation based on the qualitative contact state between the deformable workpiece and a rigid environment. First, we give an enumeration of possible contact states and discuss the main characteristics of each state. Second, we investigate the transitions which are possible between the contact states and derive criteria and conditions for each of them. Finally, we apply the concept of contact states and state transitions to the description of a typical assembly task.

Presents an approach to parallel path planning for industrial robot arms with six degrees of freedom in an online given 3D environment. The method is based a best-first search algorithm and needs no essential off-line computations. The algorithm works in an implicitly discrete configuration space. Collisions are detected in the Cartesian workspace by hierarchical distance computation based on polyhedral models of the robot and the obstacles. By decomposing the 6D configuration space into hypercubes and cyclically mapping them onto multiple processing units, a good load distribution can be achieved. We have implemented the parallel path planner on a workstation cluster with 9 PCs and tested the planner for several benchmark environments. With optimal discretisation, the new approach usually shows very good speedups. In online provided environments with static obstacles, the parallel planning times are only a few seconds.

The paper is based on a path planning approach for industrial robot arms with 6 degrees of freedom in an online given 3D environment. It has online capabilities by searching in an implicit and discrete configuration space and detecting collisions in the Cartesian workspace by distance computation based on the given CAD model. Here, we present different methods for specifying the C-space discretization. Besides the usual uniform and heuristic discretization, we investigate two versions of an optimal discretization for a user-predefined Cartesian resolution. The different methods are experimentally evaluated. Additionally, we provide a set of 3-dimensional benchmark problems for a fair comparison of the path planner. For each benchmark, the run-times of our planner are between only 3 and 100 seconds on a Pentium PC with 133 MHz.

One of the many features needed to support the activities of autonomous systems is the ability of motion planning. It enables robots to move in their environment securely and to accomplish given tasks. Unfortunately, the control loop comprising sensing, planning, and acting has not yet been closed for robots in dynamic environments. One reason involves the long execution times of the motion planning component. A solution for this problem is offered by the use of highly computational parallelism. Thus, an important task is the parallelization of existing motion planning algorithms for robots so that they are suitable for highly computational parallelism. In several cases, completely new algorithms have to be designed, so that a parallelization is feasible. We review approaches to motion planning using parallel computation.

A fast method for computing the collision vector for online collision detection with a multi-arm robot is presented. Manipulators and obstacles are modeled by sets of convex polytopes. Known distance algorithms serve as a foundation. To speed up the collision detection dynamic obstacles are approximated by geometric primitives and organized in hierarchies. Online, the dynamic hierarchies are adjusted to the current arm configuration. A comparison with previous methods showed an increased acceleration of the computations.>