For robots to solve real world tasks, they often require the ability to reason about both symbolic and geometric knowledge. We present a framework, called KABouM, for integrating knowledge-level task planning and motion planning in a bounding geometry. By representing symbolic information at the knowledge level, we can model incomplete information, sensing actions and information gain; by representing all geometric entities--objects, robots and swept volumes of motions--by sets of convex polyhedra, we can efficiently plan manipulation actions and raise reasoning about geometric predicates, such as collisions, to the symbolic level. At the geometric level, we take advantage of our bounded convex decomposition and swept volume computation with quadratic convergence, and fast collision detection of convex bodies. We evaluate our approach on a wide set of problems using real robots, including tasks with multiple manipulators, sensing and branched plans, and mobile manipulation.

We discuss the architecture and software engineering principles of the Robotics Library (RL). Driven by requirements of robot systems, research projects, industrial applications, and education, we identify relevant design requirements and present an approach to manage hardware and real-time, provide a user-friendly, object-oriented interface to powerful kinematics and dynamics calculations, and support various platforms. After over ten years of development that started in 2004 and evaluating many variants of the architecture, we discuss the design choices for the components of the library in its current version.

In this paper, we propose a framework for prioritized constraint-based specification of robot tasks. This framework is integrated with a cognitive robotic system based on semantic models of processes, objects, and workcells. The target is to enable intuitive (re-)programming of robot tasks, in a way that is suitable for non-expert users typically found in SMEs. Using CAD semantics, robot tasks are specified as geometric inter-relational constraints. During execution, these are combined with constraints from the environment and the workcell, and solved in real-time. Our constraint model and solving approach supports a variety of constraint functions that can be non-linear and also include bounds in the form of inequalities, e.g., geometric inter-relations, distance, collision avoidance and posture constraints. It is a hierarchical approach where priority levels can be specified for the constraints, and the nullspace of higher priority constraints is exploited to optimize the lower priority constraints. The presented approach has been applied to several typical industrial robotic use-cases to highlight its advantages compared to other state-of-the-art approaches.

For robots to solve hard tasks in real-world manufacturing and service contexts, they need to reason about both symbolic and geometric preconditions, and the effects of complex actions. We use an existing Knowledge of Volumes approach to robot task planning (KVP), which facilitates hybrid planning with symbolic actions and continuous-valued robot and object motion, and make two important additions to this approach: (i) new geometric predicates are added for complex object manipulation planning, and (ii) all geometric queries-such as collision and inclusion of objects and swept volumes-are implemented with a single-sided, bounded approximation, which calculates efficient and safe robot motion paths. Our task planning framework is evaluated in multiple scenarios, using concise and generic scenario definitions.

For intelligent robots to solve real-world tasks, they need to manipulate multiple objects, and perform diverse manipulation actions apart from rigid transfers, such as pushing and sliding. Planning these tasks requires discrete changes between actions, and continuous, collision-free paths that fulfill action-specific constraints. In this work, we propose a multi-modal path planner, named MOPL, which accepts generic definitions of primitive actions with different types of contact manifolds, and randomly spans its search trees through these subspaces. Our evaluation shows that this generic search technique allows MOPL to solve several challenging scenarios over different types of kinematics and tools with reasonable performance. Furthermore, we demonstrate MOPL by solving and executing plans in two real-world experimental setups.

We present a novel approach of active object categorization based on an iterative Bayesian method using capacitive sensing during object manipulation. The approach uses a novel type of capacitive sensor, which can measure internal properties of materials that are inaccessible to vision or tactile sensing. The electrodes of this capacitive sensor are sufficiently flexible and thin to be attached to various types of robot tools, including humanoid fingers and palms. They are mechanically robust and wear resistant. In comparison to earlier capacitive sensors systems, we perform single ended and differential measurements over an array of electrodes, and coordinate measurement with robot manipulation to extract information not available from static measurements. We demonstrate the capability of our active object categorization system in the James robot bartender system. This system can manipulate objects and measure continuously in order to categorize empty and non-empty bottles. The principle of capacitive sensing during manipulation can be applied to more general object manipulation tasks in robotics and also in other fields of industrial automation.

In this paper, we propose a framework for intuitive task-based programming of robots using geometric inter-relational constraints. The intended applications of this framework are robot programming interfaces that use semantically rich task descriptions, allow intuitive (re-)programming, and are suitable for non-expert users typically found in SMEs. A key concept in this work is the use of CAD semantics to represent geometric entities in the robotic workcell. The robot tasks are then represented as a set of geometrical inter-relational constraints, which are solved in real-time to be executed on the robot. Since these constraints often specify the target pose only partially, the robot can be controlled to move in the constraints' null space in order to handle external disturbances or further optimize the robot's pose during runtime. Geometrical inter-relational constraints are easy to understand and can be intuitively specified using CAD software. A number of applications common in industrial robotic scenarios have been chosen to highlight the advantages of the presented approach vis-à-vis the state-of-the-art approaches.

A hybrid planning approach is presented in this paper with the focus of integrating task planning and motion planning for highly automated driving. In the context of task planning, the vehicle and environment states are transformed from the continuous configuration space to a discrete state space. A planning problem is solved by a search algorithm for an optimal task sequence to reach the goal conditions in the symbolic space, regarding constraints such as space topology, place occupation, and traffic rules. Each task can be mapped to a specific driving maneuver and solved with a dedicated motion planning method in the continuous configuration space. The task planning approach not only bridges the gap between high-level navigation and low-level motion planning, but also provides a modular domain description that can be developed and verified individually. Our task planner for automated driving is evaluated in several scenarios with prior knowledge about the road-map and sensing range of the vehicle. Behavior that is otherwise complex to achieve is planned according to traffic rules and re-planned regarding the on-line perception.

This paper deals with recognizing human actions in depth video data. Current state-of-the-art action recognition methods use hand-designed features, which are difficult to produce and time-consuming to extend to new modalities. In this paper, we propose a novel, 3.5D representation of a depth video for action recognition. A 3.5D graph of the depth video consists of a set of nodes that are the joints of the human body. Each joint is represented by a set of spatio-temporal features, which are computed by an unsupervised learning approach. However, if occlusions occur, the 3D positions of the joints are noisy which increases the intra-class variations in action classes. To address this problem, we propose the Ensemble Weighted Multi-Instance Learning approach (EnwMi) for the action recognition task. It considers the class imbalance and intra-class variations. We formulate the action recognition task with depth videos as a weighted multi-instance problem. We further integrate an ensemble learning method into the weighted multi-instance learning framework. Our approach is evaluated on Microsoft Research Action3D dataset, and the results show that it outperforms state-of-the-art methods.

Today's industrial robots require expert knowledge and are not profitable for small and medium sized enterprises with their small lot sizes. It is our strong belief that more intuitive robot programming in an augmented reality robot work cell can dramatically simplify re-programming and leverage robotics technology in short production cycles. In this paper, we present a novel augmented reality system for defining virtual obstacles, specifying tool positions, and specifying robot tasks. We evaluate the system in a user study and, more specifically, investigate the input of robot end-effector orientations in general.

In this paper we present results from a user evaluation of a robot bartender system which handles state uncertainty derived from speech input by using belief tracking and generating appropriate clarification questions. We present a combination of state estimation and action selection components in which state uncertainty is tracked and exploited, and compare it to a baseline version that uses standard speech recognition confidence score thresholds instead of belief tracking. The results suggest that users are served fewer incorrect drinks when the uncertainty is retained in the state.

A robot agent existing in the physical world must be able to understand the social states of the human users it interacts with in order to respond appropriately. We compared two implemented methods for estimating the engagement state of customers for a robot bartender based on low-level sensor data: a rule-based version derived from the analysis of human behaviour in real bars, and a trained version using supervised learning on a labelled multimodal corpus. We first compared the two implementations using cross-validation on real sensor data and found that nearly all classifier types significantly outperformed the rule-based classifier. We also carried out feature selection to see which sensor features were the most informative for the classification task, and found that the position of the head and hands were relevant, but that the torso orientation was not. Finally, we performed a user study comparing the ability of the two classifiers to detect the intended user engagement of actual customers of the robot bartender; this study found that the trained classifier was faster at detecting initial intended user engagement, but that the rule-based classifier was more stable.

We address the question of whether service robots that interact with humans in public spaces must express socially appropriate behaviour. To do so, we implemented a robot bartender which is able to take drink orders from humans and serve drinks to them. By using a high-level automated planner, we explore two different robot interaction styles: in the task only setting, the robot simply fulfils its goal of asking customers for drink orders and serving them drinks; in the socially intelligent setting, the robot additionally acts in a manner socially appropriate to the bartender scenario, based on the behaviour of humans observed in natural bar interactions. The results of a user study show that the interactions with the socially intelligent robot were somewhat more efficient, but the two implemented behaviour settings had only a small influence on the subjective ratings. However, there were objective factors that influenced participant ratings: the overall duration of the interaction had a positive influence on the ratings, while the number of system order requests had a negative influence. We also found a cultural difference: German participants gave the system higher pre-test ratings than participants who interacted in English, although the post-test scores were similar.

Robot task planning is an inherently challenging problem, as it covers both continuous-space geometric reasoning about robot motion and perception, as well as purely symbolic knowledge about actions and objects. This paper presents a novel “knowledge of volumes” framework for solving generic robot tasks in partially known environments. In particular, this approach (abbreviated, KVP) combines the power of symbolic, knowledge-level AI planning with the efficient computation of volumes, which serve as an intermediate representation for both robot action and perception. While we demonstrate the effectiveness of our framework in a bimanual robot bartender scenario, our approach is also more generally applicable to tasks in automation and mobile manipulation, involving arbitrary numbers of manipulators.

All mammals but humans use whiskers in order to rapidly acquire information about objects in the vicinity of the head. Collisions of the head and objects can be avoided as the contact point is moved from the body surface to the whiskers. Such a behavior is also highly desirable during many robot tasks such as for human-robot interaction. Using novel capacitive proximity sensors, robots sense when they approach a human (or an object) and react before they actually collide with it. We propose a sensor and control concept that mimics the behavior of whiskers by means of capacitive sensors. Major advantages are the absence of physical whiskers, the absence of blind spots and a very short response time. The sensors are flexible and thin so that they feature skin-like properties and can be attached to various robotic link and joint shapes. In comparison to capacitive proximity sensors, the proposed virtual whiskers offer better sensitivity towards small conductive as well as non conductive objects. Equipped with the new proximity sensors, a seven-joint robot for humanrobot interaction tasks shows the efficiency and responsiveness of our concept.

We introduce a humanoid robot bartender that is capable of dealing with multiple customers in a dynamic, multi-party social setting. The robot system incorporates state-of-the-art components for computer vision, linguistic processing, state management, high-level reasoning, and robot control. In a user evaluation, 31 participants interacted with the bartender in a range of social situations. Most customers successfully obtained a drink from the bartender in all scenarios, and the factors that had the greatest impact on subjective satisfaction were task success and dialogue efficiency.

Robots that interact with humans in everyday situations, need to be able to interpret the nonverbal social cues of their human interaction partners. We show that humans use body posture and head pose as social signals to initiate and terminate interaction when ordering drinks at a bar. For that, we record and analyze 108 interactions of humans interacting with a human bartender. Based on these findings, we train a Hidden Markov Model (HMM) using automatic body posture and head pose estimation. With this model, the bartender robot of the project JAMES can recognize typical social behaviors of human customers. Evaluation shows a recognition rate of 82.9 % for all implemented social behaviors and in particular a recognition rate of 91.2 % for bartender attention requests, which will allow the robot to interact with multiple humans in a robust and socially appropriate way.

The control of tendon-driven and, in particular, of anthropomimetic robots using techniques from traditional robotics remains a very challenging task [1, 2]. Hence, we previously proposed to employ physics-based simulation engines to simulate the complex dynamics of this emerging class of robots [3] and to use the simulation model as an internal model for robot control [4]. This approach, however, relies on an accurate model to be successful. In this paper, we present the automated, steady-state pose calibration of a physics-based, anthropomimetic robot model using a (μ, λ)-Evolution Strategy. For the acquisition of the poses of the physical robot, a stereo-vision, infrared-marker based motion capture system with real-time capabilities was developed. The employed (μ, λ)-Evolution Strategy uses a Gaussian-based, non-isotropic, self-adapting mutation operator to explore the search space and reduce the simulation-reality gap. The obtained results are impressive, resulting in a reduction of joint angle errors in the range of one to two orders of magnitude and an absolute joint angle error of 0.5°-4.5° per pose evaluated.

This paper presents a novel approach based on Random Finite Set (RFS) Statistics for estimating a vehicle's trajectory in complex urban environments by using a fixed single camera. For this, we extend our earlier works which used Probability Hypothesis Density (PHD) filtering under sensor fusion framework and are among the first to apply this technique to visual odometry in real traffic scenes. We consider features acquired from the camera as a group targets, use the PHD filter to update the overall group state and then estimate the ego-motion vector of the camera. Compared to other approaches, our approach presents a recursive filtering algorithm that provides dynamic estimation of multiple-targets states in the presence of clutter and avoids the association problem. Experimental results show that this method provides good robustness under real traffic scenarios.

We present a general framework for tracking image regions in two views simultaneously based on sum-of-squared differences (SSD) minimization. Our method allows for motion models up to affine transformations. Contrary to earlier approaches, we incorporate the well-known epipolar constraints directly into the SSD optimization process. Since the epipolar geometry can be computed from the image directly, no prior calibration is necessary. Our algorithm has been tested in different applications including camera localization, wide-baseline stereo, object tracking and medical imaging. We show experimental results on robustness and accuracy compared to the known ground truth given by a conventional tracking device.