Learning-based methods excel at robot motion generation but remain limited in contact-rich physical interaction. Impedance control provides stable and safe contact behavior but requires task-specific tuning of stiffness and damping parameters. We present Diffusion-Based Impedance Learning, a framework that bridges these paradigms by combining generative modeling with energy-consistent impedance control. A Transformer-based Diffusion Model, conditioned via cross-attention on measured external wrenches, reconstructs simulated Zero-Force Trajectories (sZFTs) that represent contact-consistent equilibrium behavior. A SLERP-based quaternion noise scheduler preserves geometric consistency for rotations on the unit sphere. The reconstructed sZFT is used by an energy-based estimator to adapt impedance online through directional stiffness and damping modulation. Trained on parkour and robot-assisted therapy demonstrations collected via Apple Vision Pro teleoperation, the model achieves sub-millimeter positional and sub-degree rotational accuracy using only tens of thousands of samples. Deployed in real-time torque control on a KUKA LBR iiwa, the approach enables smooth obstacle traversal and generalizes to unseen tasks, achieving 100% success in multi-geometry peg-in-hole insertion. The code for all experiments is publicly available on GitHub and videos of the experiments are available on the project website.

By incorporating physical consistency as inductive bias, deep neural networks display increased generalization capabilities and data efficiency in learning nonlinear dynamic models. However, the complexity of these models generally increases with the system dimensionality, requiring larger datasets, more complex deep networks, and significant computational effort. We propose a novel geometric network architecture to learn physically-consistent reduced-order dynamic parameters that accurately describe the original high-dimensional system behavior. This is achieved by building on recent advances in model-order reduction and by adopting a Riemannian perspective to jointly learn a non-linear structure-preserving latent space and the associated low-dimensional dynamics. Our approach enables accurate long-term predictions of the high-dimensional dynamics of rigid and deformable systems with increased data efficiency by inferring interpretable and physically-plausible reduced Lagrangian models.

Accurately modeling and predicting complex dynamical systems, particularly those involving force exchange and dissipation, is crucial for applications ranging from fluid dynamics to robotics, but presents significant challenges due to the intricate interplay of geometric constraints and energy transfer. This paper introduces Geometric Contact Flows (GFC), a novel framework leveraging Riemannian and Contact geometry as inductive biases to learn such systems. GCF constructs a latent contact Hamiltonian model encoding desirable properties like stability or energy conservation. An ensemble of contactomorphisms then adapts this model to the target dynamics while preserving these properties. This ensemble allows for uncertainty-aware geodesics that attract the system’s behavior toward the data support, enabling robust generalization and adaptation to unseen scenarios. Experiments on learning dynamics for physical systems and for controlling robots on interaction tasks demonstrate the effectiveness of our approach.

Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. While there exist several real-world bimanual systems, there is a lack of simulated benchmarks with a large task diversity for systematically studying bimanual capabilities across a wide range of tabletop tasks. This paper addresses the gap by presenting a benchmark for bimanual manipulation. A key functionality is the ability to autonomously generate training data without the necessity of human demonstrations to the robot. We open-source our code and benchmark, which comprises 13 new tasks with 23 unique task variations, each requiring a high degree of coordination and adaptability. To initiate the benchmark, we extended multiple state-of-the-art techniques to the domain of bimanual manipulation. The project website with code is available at: http://bimanual.github.io.

The online adaptation of exoskeleton control based on muscle activity sensing offers a promising approach to personalizing exoskeleton behavior based on the user's biosignals. While electromyography (EMG)-based methods have demonstrated improvements in joint torque estimation, EMG sensors require direct skin contact and extensive post-processing. In contrast, force myography (FMG) measures normal forces resulting from changes in muscle volume due to muscle activity. We propose an FMG-based method to estimate knee and ankle joint torques by integrating joint angles and velocities with muscle activity data. We learn a model for joint torque estimation using Gaussian process regression (GPR). The effectiveness of the proposed FMG-based method is validated on isokinetic motions performed by ten participants. The model is compared to a baseline model that uses only joint angle and velocity as well as a model augmented by EMG data. The results indicate that incorporating FMG into exoskeleton control can improve the estimation of joint torque for the ankle and knee joints in novel task characteristics within a single participant. Although the findings suggest that this approach may not improve the generalizability of estimates between multiple participants, they highlight the need for further research into its potential applications in exoskeleton control.

Humanoid robotic hands need to be versatile and capable of providing environmental information in order to serve as a platform for intelligent grasp control. To facilitate the design process of such hands, we present the KIT Robotic Hands. They have been designed to meet diverse application requirements through their scalability in size, actuation, sensorization and computing resources. The hands integrate a multi-modal sensor system, in-hand embedded processing capabilities, an adaptive underactuated mechanism and a continuously controllable thumb rotation to enhance dexterity. The flexibility of the design is demonstrated through two application-specific hand implementations: one is the ARMAR-7 hand, which has human hand dimensions for grasping daily objects in household tasks, the other is the ARMAR-DE hand, a larger hand designed for grasping bigger objects in decontamination tasks. We describe the design and mechatronics of the hands as well as an evaluation of the grasp success and image segmentation based on an in-hand integrated camera and onboard processing of visual data.

Tu Anh Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Human motion taxonomies serve as high-level hierarchical abstractions that classify how humans move and interact with their environment. They have proven useful to analyse grasps, manipulation skills, and whole-body support poses. Despite substantial efforts devoted to design their hierarchy and underlying categories, their use remains limited. This may be attributed to the lack of computational models that fill the gap between the discrete hierarchical structure of the taxonomy and the high-dimensional heterogeneous data associated to its categories. To overcome this problem, we propose to model taxonomy data via hyperbolic embeddings that capture the associated hierarchical structure. We achieve this by formulating a novel Gaussian process hyperbolic latent variable model that incorporates the taxonomy structure through graph-based priors on the latent space and distance-preserving back constraints. We validate our model on three different human motion taxonomies to learn hyperbolic embeddings that faithfully preserve the original graph structure. We show that our model properly encodes unseen data from existing or new taxonomy categories, and outperforms its Euclidean and VAE-based counterparts. Finally, through proof-of-concept experiments, we show that our model may be used to generate realistic trajectories between the learned embeddings.

Visual imitation learning has achieved impressive progress in learning unimanual manipulation tasks from a small set of visual observations, thanks to the latest advances in computer vision. However, learning bimanual coordination strategies and complex object relations from bimanual visual demonstrations, as well as generalizing them to categorical objects in novel cluttered scenes remain unsolved challenges. In this paper, we extend our previous work on keypoints-based visual imitation learning (K-VIL) [1] to bimanual manipulation tasks. The proposed Bi-KVIL jointly extracts so-called Hybrid Master-Slave Relationships (HMSR) among objects and hands, bimanual coordination strategies, and sub-symbolic task representations. Our bimanual task representation is object-centric, embodiment-independent, and viewpoint-invariant, thus generalizing well to categorical objects in novel scenes. We evaluate our approach in various real-world applications, showcasing its ability to learn fine-grained bimanual manipulation tasks from a small number of human demonstration videos. Videos and source code are available at https://sites.google.com/view/bi-kvil.

Movement primitives (MPs) are compact representations of robot skills that can be learned from demonstrations and combined into complex behaviors. However, merely equipping robots with a fixed set of innate MPs is insufficient to deploy them in dynamic and unpredictable environments. Instead, the full potential of MPs remains to be attained via adaptable, large-scale MP libraries. In this paper, we propose a set of seven fundamental operations to incrementally learn, improve, and re-organize MP libraries. To showcase their applicability, we provide explicit formulations of the five spatial operations for libraries composed of Via-Point Movement Primitives (VMPs). By building on Riemannian manifold theory, our approach enables the incremental learning of all parameters of position and orientation VMPs within a library. Moreover, our approach stores a fixed number of parameters, thus complying with the essential principles of incremental learning. We evaluate our approach to incrementally learn a VMP library from sequentially-provided motion capture data.

Ankle exoskeletons can assist the ankle joint and reduce the metabolic cost of walking. However, many existing ankle exoskeletons constrain the natural 3 degrees of freedom (DoF) of the ankle to limit the exoskeleton’s weight and mechanical complexity, thereby compromising comfort and kinematic compatibility with the user.This paper presents a novel ankle exoskeleton frame design that allows for 3 DoF ankle motion using a symmetric parallel frame design principle resulting in a strong frame while weighing 1.8 kg. Furthermore, a cable routing method is proposed to actuate the plantarflexion of the ankle. The kinematic compatibility of the proposed exoskeleton frame is evaluated in straight- and curve-walking scenarios with four users. The study demonstrates that the exoskeleton frame adapts to the natural 3 DoF ankle motion and the range of motion (RoM) during walking. The actuation in plantarflexion is evaluated in a stationary torque experiment demonstrating the ability of the frame to transfer large torque loads of up to 57.4 Nm. This work contributes to the design and development of more flexible and adaptable ankle exoskeletons for walking assistance.

In the realm of robotics, numerous downstream robotics tasks leverage machine learning methods for processing, modeling, or synthesizing data. Often, this data comprises variables that inherently carry geometric constraints, such as the unit-norm condition of quaternions representing rigid-body orientations or the positive definiteness of stiffness and manipulability ellipsoids. Handling such geometric constraints effectively requires the incorporation of tools from differential geometry into the formulation of machine learning methods. In this context, Riemannian manifolds emerge as a powerful mathematical framework to handle such geometric constraints. Nevertheless, their recent adoption in robot learning has been largely characterized by a mathematically-flawed simplification, hereinafter referred to as the "single tangent space fallacy". This approach involves merely projecting the data of interest onto a single tangent (Euclidean) space, over which an off-the-shelf learning algorithm is applied. This paper provides a theoretical elucidation of various misconceptions surrounding this approach and offers experimental evidence of its shortcomings. Finally, it presents valuable insights to promote best practices when employing Riemannian geometry within robot learning applications.

Generating human-like robot motions is pivotal for achieving smooth human-robot interactions. Such motions contribute to better predictions of robot motions by humans, thus leading to more intuitive interaction and increased acceptability. Human likeness in robot motions has been conventionally measured and realized via the optimization of human-likeness metrics. However, the abundance of such metrics and the absence of standardized criteria impede their usage in novel contexts. In this work, we introduce a unified human-likeness metric built from a hierarchically weighted sum of individual metrics. The proposed metric is derived from a thorough analysis of eleven existing human-likeness criteria and is applicable across various tasks and robot models. We evaluate its performance in the context of motion retargeting of bimanual tasks with three different humanoid robots.

Artificial cognitive architectures traditionally rely on complex memory models to encode, store, and retrieve information. However, the conventional practice of transferring all data from working memory (WM) to long-term memory (LTM) leads to high data volumes and challenges in efficient information processing and access. Deciding what information to retain or discard within a robot’s LTM is particularly challenging since knowledge about future data utilization is absent. Drawing inspiration from human forgetting this paper implements and evaluates novel forgetting techniques that allow consolidation in the robot’s LTM only when new information is encountered. The proposed approach combines fast filtering during data transfer to the robot’s LTM with slower yet more precise forgetting mechanisms that are periodically evaluated for offline data deletion inside the LTM. We compare different mechanisms, utilizing metrics such as data similarity, data age, and consolidation frequency. The efficacy of forgetting techniques is evaluated by comparing their performance in a task where two ARMAR robots search through their LTM for past object locations in episodic ego-centric images and robot state data. Experimental results show that our forgetting techniques significantly reduce the space requirements of a robot’s LTM while maintaining its capacity to successfully perform tasks relying on LTM information. Notably, similarity-based forgetting methods outperform frequency- and time-based approaches. The combination of online frequency-based, online similarity-based, offline similarity-based, and time-based decay methods shows superior performance compared to using individual forgetting strategies.

Grasping with five-fingered humanoid hands is a complex control problem. Throughout the entire grasping motion, all finger joints need to be coordinated to achieve a stable grasp. Grasp synergies provide a simplified, low-dimensional representation of grasp postures and motions, that can be used for the description of human grasps as well as the generation of novel, human-like grasps. However, the abstract synergy representation complicates the association of relevant high-level grasp parameters, as for example the grasp type and final posture or the grasp speed. Therefore, it is difficult to control these grasp characteristics in the synergy space. This paper presents an adaptable representation for kinematic grasping motions in synergy space, that allows the generation of novel, human-like grasps under direct control of high-level grasp parameters. It is based on via-point movement primitives trained on synergy trajectories of human grasping motions. The representation using synergy primitives allows for a straightforward adaptation of grasp characteristics while preserving the essential grasping motion learned from human demonstration. The kinematic synergy primitives have a low reproduction error of 3.9% of the maximum finger joint angle and are able to generate successful grasps on a simulated human hand and a real prosthetic hand.

We introduce Riemannian Flow Matching Policies (RFMP), a novel model for learning and synthesizing robot visuomotor policies. RFMP leverages the efficient training and inference capabilities of flow matching methods. By design, RFMP inherits the strengths of flow matching: the ability to encode high-dimensional multimodal distributions, commonly encountered in robotic tasks, and a very simple and fast inference process. We demonstrate the applicability of RFMP to both state-based and vision-conditioned robot motion policies. Notably, as the robot state resides on a Riemannian manifold, RFMP inherently incorporates geometric awareness, which is crucial for realistic robotic tasks. To evaluate RFMP, we conduct two proof-of-concept experiments, comparing its performance against Diffusion Policies. Although both approaches successfully learn the considered tasks, our results show that RFMP provides smoother action trajectories with significantly lower inference times.

Task-oriented object grasping and rearrangement are key skills for robots, which have to perform versatile real-world manipulation tasks. However, they remain challenging due to partial observations of the objects and shape variations in categorical objects. In this paper, we present the Multi-feature Implicit Model (MIMO), a novel object representation that encodes multiple spatial features between a point and an object in an implicit neural field. Training such a model on multiple features ensures that it embeds the object shapes consistently in different aspects, thus improving its performance in object shape reconstruction from partial observation, shape similarity measure, and modeling spatial relations between objects. Based on MIMO, we propose a framework to learn task-oriented object grasping and rearrangement from single or multiple human demonstration videos. The evaluations in simulation show that our approach outperforms the state-of-the-art methods for multi- and single-view observations. Real-world experiments demonstrate the efficacy of our approach in one- and few-shot imitation learning of manipulation tasks.

Advancements in Industry 4.0 demand sophisticated solutions for automatic robotic assembly sequence planning (RASP), capable of handling the diversity and complexity of modern manufacturing tasks. One approach to RASP is Assembly-by-Disassembly (AbD). It first searches for a disassembly sequence that is then inverted to obtain an assembly sequence. One of the challenges of AbD, however, is the exponential number of potential assembly sequences for any given assembly. To mitigate this challenge, we propose to transfer knowledge obtained during previous planning attempts. Specifically, we present an approach that combines Monte Carlo Tree Search (MCTS) with deep Q-learning to optimize the total length of robotic assembly paths. We use a graph-based representation of disassembly states in combination with a graph neural network to learn the Q-function. We further discuss a principled approach to generate 3D assemblies out of aluminium profiles that a single robot manipulator can assemble. With this approach, we generated two datasets consisting of 14 assemblies with 21 removable parts and 7 assemblies with 30 removable parts. Using leave-one-out cross-validation, we were able to demonstrate how our approach outperformed an unmodified MCTS. Moreover, we successfully transferred knowledge between datasets.

Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the subsymbolic level such as the duration of different actions, or their starting and end points in time. Such temporal constraints are crucial for temporal planning, reasoning, and the exact timing for the execution of bimanual actions on a bimanual robot. In our previous work, we addressed the learning of temporal task constraints on the symbolic level and demonstrated how a robot can leverage this knowledge to respond to failures during execution. In this work, we propose a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations. Our main contributions are a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints, as well as a method based on fuzzy logic to derive symbolic temporal task constraints from this representation. This complements our previous work on learning comprehensive temporal task models by integrating symbolic and subsymbolic information based on a subsymbolic foundation, while still maintaining the symbolic expressiveness of our previous approach. We compare our proposed approach with our previous pure-symbolic approach and show that we can reproduce and even outperform it. Additionally, we show how the subsymbolic temporal task constraints can synchronize otherwise unimanual movement primitives for bimanual behavior on a humanoid robot.

Executing bimanual manipulation tasks on humanoid robots introduces additional challenges due to inherent spatial and temporal coordination between both hands. In our previous work, we proposed the Bimanual Manipulation Taxonomy, which defines categories of bimanual manipulation strategies based on the coordination and physical interaction between both hands, the role of each hand in the task, and the symmetry of arm movements during task execution. In this work, we build upon this taxonomy and provide a formalization of temporal and spatial constraints associated with each category of the taxonomy. This formalization uses Petri nets to represent temporal constraints and differentiates between relative and global targets. We incorporate these constraints in a category-specific controller to enable reactive adaptation of the behavior according to the respective coordination constraints. We evaluated our approach in simulation and in real-world experiments on the humanoid robot ARMAR-6. The results demonstrate that category-specific constraints can be enforced when needed while maintaining flexibility to accommodate additional constraints.

To perform versatile mobile manipulation tasks in human-centered environments, the ability to efficiently transfer learned skills, knowledge, and experiences from one robot to another or across different environments is critical. In this paper, we present MAkEable, a versatile uni- and multi-manual mobile manipulation framework that facilitates the transfer of capabilities and knowledge across different tasks, environments, and robots. Our framework integrates an affordance-based task description into the memory-centric cognitive architecture of the ARMAR humanoid robot family, which supports the sharing of experiences and demonstrations for transferring mobile manipulation skills. By representing mobile manipulation actions through affordances, i. e., interaction possibilities of the robot with its environment, we provide a unifying framework for the autonomous uni- and multi-manual manipulation of known and unknown objects in various environments. We demonstrate MAkEable’s applicability in real-world experiments for multiple robots, tasks, and environments. This includes grasping known and unknown objects, object placing, bimanual object grasping, memory-enabled skill transfer in a drawer opening scenario across two different humanoid robots, and a pouring task learned from human demonstration. Code is available through our project page1.

Humanoid robots will be able to assist humans in their daily life, in particular due to their versatile action capabilities. However, while these robots need a certain degree of autonomy to learn and explore, they also should respect various constraints, for access control and beyond. We explore the novel field of incorporating privacy, security, and access control constraints with robot task planning approaches. We report preliminary results on the classical symbolic approach, deep-learned neural networks, and modern ideas using large language models as knowledge base. From analyzing their trade-offs, we conclude that a hybrid approach is necessary, and thereby present a new use case for the emerging field of neuro-symbolic artificial intelligence.

Industry 4.0 facilitates a high number of product variants, posing significant challenges for modern manufacturing. One of them is the automatic creation of assembly sequences. This can be achieved with the assembly-by-disassembly (AbD) approach, which is currently highly inefficient. We aim at speeding up AbD by leveraging deep learning. AbD relies on iteratively testing parts for removal, which makes the order in which parts are tested highly relevant for its run-time. We optimize this order by training a graph neural network (GNN) based on the shape of parts and the shape of local part connections. For each part, it predicts a removability probability. We use these probabilities to optimize the order in which parts are tested for removal. This reduces the number of parts tested by approximately 64%-90%, depending on the tested product. Further improvements are achieved by combining our approach with bookkeeping, another approach for speeding up AbD. Finally, we separately analyze the impact of the parts and their connections on the removability probabilities predicted by the GNN. We found that most of the important information regarding a part's removability can be derived from its connections alone.

In this paper, we present a method to engage measurement uncertainties with the probabilistic robustness to one system uncertainty measure. Providing a metric indicating the potential occurrence of dangerous situations is highly essential for safety-critical robot applications. Due to the difficulty of finding a quantifiable, unambiguous representation however, such a metric has not been derived to date. In case of sensory devices, measurement uncertainties are usually provided by manufacturer specifications. Apart from that, several contributions demonstrate that the accuracy of neural networks is verifiable via the robustness. However, state-of-the-art literature is mainly concerned with theoretical investigations such that scarce attention has been devoted to the transfer of the robustness to real-world applications. To fill this gap, we show how the probabilistic robustness can be made useful for evaluating quantitative safety limits. Our key idea is to exploit the analogy between measurement uncertainties and the probabilistic robustness: While measurement uncertainties reflect possible shifts due to technical limitations, the robustness refers to the tolerated amount of distortions in the input data for an unaltered output. Inspired by this analogy, we combine both measures to quantify the system uncertainty online. We validate our method in different settings under real-world conditions. Our findings exemplify that incorporating the novel uncertainty metric effectively prevents the rate of dangerous situations in Human-Robot Collaboration.

The generation of energy-efficient and dynamic-aware robot motions that satisfy constraints such as joint limits, self-collisions, and collisions with the environment remains a challenge. In this context, Riemannian geometry offers promising solutions by identifying robot motions with geodesics on the so-called configuration space manifold. While this manifold naturally considers the intrinsic robot dynamics, constraints such as joint limits, self-collisions, and collisions with the environment remain overlooked. In this paper, we propose a modification of the Riemannian metric of the configuration space manifold allowing for the generation of robot motions as geodesics that efficiently avoid given regions. We introduce a class of Riemannian metrics based on barrier functions that guarantee strict region avoidance by systematically generating accelerations away from no-go regions in joint and task space. We evaluate the proposed Riemannian metric to generate energy-efficient, dynamic-aware, and collision-free motions of a humanoid robot as geodesics and sequences thereof.

Humans naturally execute many everyday manipulation actions with both arms simultaneously. Similarly, endowing robots with bimanual manipulation task models is key to efficiently perform complex manipulation tasks. To do so, a promising approach is to learn a library of task models from human demonstrations. However, this requires human motions to be meaningfully segmented. In this paper, we propose to segment the motion of each hand individually to account for different bimanual coordination patterns and provide a thorough evaluation of state-of-the-art segmentation algorithms on bimanual manipulation datasets. In particular, we compare segmentation algorithms at trajectory and semantic level with hierarchical algorithms. Moreover, our evaluation extensively studies the performances of various segmentation algorithms over a novel extension of the KIT Bimanual Manipulation Dataset featuring ~ 176 minutes of human motion recordings in household scenarios.

Obtaining reliable detections of a human is crucial for many safety-related robotic tasks. This can be done by human pose estimation methods, which predict the position of several different keypoints of the human body. In most cases, recent approaches based on neural networks produce ‘good’ results, i.e. predictions with small localization errors, however, large errors do also occur. For an individual keypoint prediction, the magnitude of the error is unknown, posing a risk to safety. In this work, we extend a neural network architecture for single-person 2D human pose estimation, so that it predicts not only the keypoints of the human body, but also corresponding upper bounds for their localization errors. These upper bounds correspond to the neural network's confidence in its output, and are obtained by one of two general strategies based on (i) a direct estimation of the localization error or (ii) the predicted standard deviations of a 2D Gaussian. We propose several approaches employing these strategies and evaluate them on the MPII Human Pose dataset. In addition, we consider two quality criteria for the results: closeness of the predicted keypoint position to the actual one, and closeness of the predicted upper bound to the localization error. The best results are achieved by a Gaussian-based approach, which predicted correct upper bounds in 94.7% of the cases, while also sufficiently fulfilling the quality criteria.

Humanoid robots will be able to assist humans in their daily life, in particular due to their versatile action capabilities. However, while these robots need a certain degree of autonomy to learn and explore, they also should respect various constraints, for access control and beyond. We explore incorporating privacy and security constraints (Activity-Centric Access Control and Deep Learning Based Access Control) with robot task planning approaches (classical symbolic planning and end-to-end learning-based planning). We report preliminary results on their respective trade-offs and conclude that a hybrid approach will most likely be the method of choice.

Visual imitation learning provides efficient and intuitive solutions for robotic systems to acquire novel manipulation skills. However, simultaneously learning geometric task constraints and control policies from visual inputs alone remains a challenging problem. In this article, we propose thekeypoint-based visual imitation learning(K-VIL) approach that automatically extracts sparse, object-centric, and embodiment-independent task representations from a small number of human demonstration videos. The task representation is composed of keypoint-based geometric constraints on principal manifolds, their associated local frames, and the movement primitives that are then needed for the task execution. Our approach is capable of extracting such task representations from a single-demonstration video and of incrementally updating them when new demonstrations are available. To reproduce manipulation skills using the learned set of prioritized geometric constraints in novel scenes, we introduce a novel keypoint-based admittance controller. We evaluate our approach in several real-world applications, showcasing its ability to deal with cluttered scenes, viewpoint mismatch, new instances of categorical objects, and large object pose and shape variations. Our evaluation demonstrates the efficiency and robustness of our approach in both one-shot and few-shot imitation learning settings.

Grasping is the process of picking up an object by applying forces and torques at a set of contacts. Recent advances in deep learning methods have allowed rapid progress in robotic object grasping. In this systematic review, we surveyed the publications over the last decade, with a particular interest in grasping an object using all six degrees of freedom of the end-effector pose. Our review found four common methodologies for robotic grasping: sampling-based approaches, direct regression, reinforcement learning, and exemplar approaches In addition, we found two “supporting methods” around grasping that use deep learning to support the grasping process, shape approximation, and affordances. We have distilled the publications found in this systematic review (85 papers) into ten key takeaways we consider crucial for future robotic grasping and manipulation research.

For a robot to perform a grasping and manipulation task, it has to determine possible robot placements in the workspace, from which target objects or environmental elements relevant to the given task are reachable. This work presents a novel approach for finding placements for the mobile base of a humanoid robot in an unknown environment with multiple support planes. We propose a novel type of reachability map - the Oriented Surface Reachability Map - that takes inclined surfaces in the environment into account and has the same complexity as reachability maps designed for flat surfaces. The resulting robot placements are not limited to SE(2) but can be applied to arbitrarily oriented planes in 3D space. The proposed method was evaluated in simulation and on the humanoid robot ARMAR-6 in real-world grasping experiments. The results show that a placement can be found for over 80% of the poses that are reachable in complicated, simulated environments, with only a small runtime overhead.

Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this work, we develop SpeedFolding, a reliable and efficient bimanual system, which given user-defined instructions as folding lines, manipulates an initially crumpled garment to (1) a smoothed and (2) a folded configuration. Our primary contribution is a novel neural network architecture that is able to predict pairs of gripper poses to parameterize a diverse set of bimanual action primitives. After learning from 4300 human- annotated and self-supervised actions, the robot is able to fold garments from a random initial configuration in under 120 s on average with a success rate of 93 %. Real-world experiments show that the system is able to generalize to unseen garments of different color, shape, and stiffness. While prior work achieved 3–6 Folds Per Hour (FPH), SpeedFolding achieves 30–40 FPH. See https://pantor.github.io/speedfolding for code, videos, and datasets.

Learning temporal relations between actions in a bimanual manipulation task is important for capturing the constraints of actions required to achieve the task's goal. However, given several demonstrations of a bimanual manipulation task, the problem of identifying the true temporal dependencies between actions - if there are any - is very challenging due to contradictions. We propose a model-driven approach for learning temporal task models from multiple bimanual human demonstrations that represents temporal relations on two levels. First, temporal relations between sets of actions that exhibit a tight temporal coupling, and second, temporal relations between these sets of actions. We build on Allen's interval algebra as a representation to express relations between temporal intervals. Semantically defining these interval relations allows us to soften their formulation to deal with inaccuracies in real data obtained when observing humans demonstrating the task. Our temporal task models can be learned incrementally from multiple modalities, and allow us to reason about viable alternatives during task execution in case of unexpected events. We evaluated the approach quantitatively on two datasets and qualitatively on a humanoid robot. The evaluation shows how inherent properties of bimanual human manipulation tasks can be exploited to derive a model useful for the reproduction by humanoid robots.

The ability to detect failure during task execution and to recover from failure is vital for autonomous robots performing tasks in previously unknown environments. In this paper, we present an approach for failure detection during the execution of grasping and mobile manipulation tasks by a humanoid robot. The approach combines multi-modal sensory information consisting of proprioceptive, force and visual information to learn task models from multiple successful task executions, in order to detect failures and to externalize them for humans in an interpretable way. To this end, we define symbolic action predicates based on multi-modal sensory information to allow high-level state estimation based on action-specific decision trees. To allow symbolic failure detection, we then learn task models that are represented as Markov chains. We evaluated the approach in several pick-and-place and mobile manipulation tasks performed by a humanoid robot in a decommissioning and a household scenario. The evaluation shows that the learned task models are capable of detecting failure with an F1-score of 93 %.

Building humanoid robots with properties similar to those of humans in terms of strength and agility is a great and unsolved challenge. This work introduces a compact and lightweight wrist joint mechanism that is singularity-free and has large range of motion. The mechanism provides two degrees of freedom (DoF) and was developed for integration into a human scale humanoid robot arm. It is based on a parallel mechanism with rolling contact joint behaviour and remote actuation that facilitates a compact design with low mass and inertia. The mechanism's kinematics along with a solution of the inverse kinematics problem for the specific design, and the manipulability analysis are presented. The first prototype of the proposed mechanism shows the possible integration of actuation, sensing and electronics in small and narrow space. Experimental evaluations shows that the design feature unique performance regarding weight, speed, payload and accuracy.

Dynamic motions of humans and robots are widely driven by posture-dependent nonlinear interactions between their degrees of freedom. However, these dynamical effects remain mostly overlooked when studying the mechanisms of human movement generation. Inspired by recent works, we hypothesize that human motions are planned as sequences of geodesic synergies, and thus correspond to coordinated joint movements achieved with piecewise minimum energy. The underlying computational model is built on Riemannian geometry to account for the inertial characteristics of the body. Through the analysis of various human arm motions, we find that our model segments motions into geodesic synergies, and successfully predicts observed arm postures, hand trajectories, as well as their respective velocity profiles. Moreover, we show that our analysis can further be exploited to transfer arm motions to robots by reproducing individual human synergies as geodesic paths in the robot configuration space.

In the not too distant future, assistive humanoid robots will provide versatile assistance for coping with everyday life. In their interactions with humans, not only safety, but also security and privacy issues need to be considered. In this Blue Sky paper, we therefore argue that it is time to bring task planning and execution as a well-established field of robotics with access and usage control in the field of security and privacy closer together. In particular, the recently proposed activity-based view on access and usage control provides a promising approach to bridge the gap between these two perspectives. We argue that humanoid robots provide for specific challenges due to their task-universality and their use in both, private and public spaces. Furthermore, they are socially connected to various parties and require policy creation at runtime due to learning. We contribute first attempts on the architecture and enforcement layer as well as on joint modeling, and discuss challenges and a research roadmap also for the policy and objectives layer. We conclude that the underlying combination of decentralized systems' and smart environments' research aspects provides for a rich source of challenges that need to be addressed on the road to deployment.

Intelligent, semi-autonomous prostheses take ad-vantage of combining autonomous functions and traditional myoelectric control. With the help of visual and environment sensors, intelligent prostheses achieve a level of autonomy which relieves the user from generating elaborate electromyographic (EMG) signals for grasp type and trajectory. To achieve the desired functionality, the semi-autonomous prosthesis must efficiently process the incoming environmental data at a high rate, with low power and high accuracy. In this paper, we propose Binary-LoRAX, a low-latency runtime adaptable classifier for the semi-autonomous grasping task of prosthetic hands. We offload the classification task to an efficient binary neural network accelerator which performs high-throughput XNOR operations on digital signal processing (DSP) blocks. To tailor the classifier’s performance to the current application scenario, we propose a frequency scaling approach which dynamically switches between two modes of operation, high-performance and power-saving. At high-performance, classifications are performed with a low latency of 0.45ms, high-throughput of 4999 FPS and power consumption of ∼ 2.15 W. This enables functions such as object localization and batch classification. Switching to power-saving mode, a latency of 80 ms is maintained, with up to 19% improved classifier battery-life. Our prototypes achieve a high accuracy of up to 99.82% on a 25 class problem from the YCB graspable object dataset.

Increasing the robustness of grasping actions and the recovery from failure is key to improving a robot’s autonomy. Endowing robots with the ability to robustly grasp and manipulate unknown difficult objects such as stones is required for sample collection in unknown environments. In this paper, we present a complete system for robust grasping of stones, which integrates stone segmentation based on depth information, the generation of grasp hypotheses and pushing actions as well as their execution. In particular, our system has been designed to solve these tasks on robots with limited computing resources. We evaluate the performance in real robot experiments in the context of stone sample collection. The results show that such a challenging task is achievable under computing resource constraints.

We introduce a multi-functional robotic gripper equipped with a set of actions required for disassembly of electromechanical devices. The gripper consists of a robot arm with 5 degrees of freedom (DoF) for manipulation and a jaw gripper with a 1-DoF rotation joint and a 1-DoF closing joint. The system enables manipulation in 7 DoF and offers the ability to reposition objects in hand and to perform tasks that usually require bimanual systems. The sensor system of the gripper includes relative and absolute joint encoders, force and pressure sensors to provide feedback about interaction forces, a tool- mounted camera for screw detection and precise placement of the tool tip using image-based visual servoing. We present a data-driven method for estimating joint torques based on the output voltage and motor speed. Further, we provide methods for teaching disassembly actions based on human demonstration, their representation as movement primitives and execution based on sensory feedback. We provide quantitative results regarding positioning and torque estimation accuracy, disassembly success rate and qualitative results regarding the successful disassembly of hard disc drives.

We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding high bandwidth raw data streaming via the robots real-time data bus. The hardware acceleration allows fast detection and localization of objects based on finger-camera images and provides input for a grasping controller. To this end, we implement a resource-aware encoder-decoder Convolutional Neural Network (CNN) for pixel-wise object segmentation and run inference on the in-hand embedded system at 3.58 GOPS. We evaluate the system, consisting of the soft hand with in-finger vision and the in-hand FPGA-accelerated CNN in several experiments on the humanoid robot ARMAR-6. Specifically, we evaluate the overall system response time, the ability to perform precision grasps and test reactivity and reliability that are required for handover actions. We obtain an overall system response time of 154 ms for catching a falling object and obtain a success rate of 90 % reliability for the power drill handover tasks. Further, we successfully demonstrate ability of dexterous grasping and manipulation of a pencil from a cup.

Humans can intuitively grasp objects of different shape and weight. Throughout the grasp execution they control and coordinate the grasp forces at all contact points between the hand and the object to achieve a stable grasp. Dexterous grasping with humanoid hands relies on the perfect coordination between grasp posture and force balance at the contact points in a high dimensional space and remains a challenge. In this paper, we present temporal force synergies describing the change in human grasp forces during the grasp execution in a low-dimensional space based on two new grasp synergy models: 1) static force synergies that are derived by a Principal Component Analysis and represent temporal grasp forces as a sequence of time-independent synergy configurations and 2) dynamic force synergies that are learned by a recurrent neural network and encode the temporal change of grasp forces throughout grasp execution in a latent synergy space clustered by grasp types. We show that both synergy spaces encode human grasp forces with an error of less than 2% and allow the generation of human-like grasp force patterns. Grasp forces for stable grasps described by the dynamic force synergies achieve a grasp quality comparable to demonstrated human grasps in simulation.

Engineering humanoid robot hands with the ability to dexterously grasp objects of different sizes, shapes, mate-rial properties and weights requires sophisticated tactile sensing and intelligent controllers able to interpret sensory information and adapt contact forces with the object to achieve a stable and safe grasp. In this paper, we present a new soft humanoid hand equipped with a multimodal sensor system in each finger and a human-inspired grasp-phases controller that is able to detect the different phases of a grasping and manipulation task, adapt interaction forces with the manipulated object and balance the force distribution in both precision and power grasps based on tactile feedback. To evaluate the controller, we conducted experiments with the hand on the humanoid robot ARMAR-6 and 31 different soft and rigid everyday objects and food items with weights ranging from 4.8 g of a paper cup to 1133.8 g of a bottle, different shapes and material properties. The results show that grasping force can be reduced by 65% compared to a naive grasping approach using maximum force for grasping and manipulating both fragile objects without destruction as well as heavy objects.

Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipulation, involving interactions between rigid objects and a deformable bag. The dataset incorporates a rich variety of scenarios including different object sizes, object numbers and manipulation actions. We approach dynamics learning by proposing an object-centric graph representation and two modules which are Active Prediction Module (APM) and Position Prediction Module (PPM) based on graph neural networks with an encode-process-decode architecture. At the inference stage, we build a two-stage model based on the learned modules for single time step prediction. We combine modules with different prediction horizons into a mixed-horizon model which addresses long-term prediction. In an ablation study, we show the benefits of the two-stage model for single time step prediction and the effectiveness of the mixed-horizon model for long-term prediction tasks. Supplementary material is available at https://github.com/wengzehang/deformable_rigid_interaction_prediction

Understanding the effects of actions is essential for planning and executing robot tasks. By imagining possible action consequences, a robot can choose specific action parameters to achieve desired goal states. We present an approach for parametrizing pushing actions based on learning internal prediction models. These pushing actions must fulfill constraints given by a high-level planner, e. g., after the push the brown box must be to the right of the orange box. In this work, we represent the perceived scenes as object-centric graphs and learn an internal model, which predicts object pose changes due to pushing actions. We train this internal model on a large synthetic data set, which was generated in simulation, and record a smaller data set on the real robot for evaluation. For a given scene and goal state, the robot generates a set of possible pushing action candidates by sampling the parameter space and then evaluating the candidates by internal simulation, i. e., by comparing the predicted effect resulting from the internal model with the desired effect provided by the high-level planner. In the evaluation, we show that our model achieves high prediction accuracy in scenes with a varying number of objects and, in contrast to state-of-the-art approaches, is able to generalize to scenes with more objects than seen during training. In experiments on the humanoid robot ARMAR-6, we validate the transfer from simulation and show that the learned internal model can be used to manipulate scenes into desired states effectively.

We present a novel underactued humanoid five finger soft hand, the KIT Finger-Vision Soft Hand, which is equipped with cameras in the fingertips and integrates a high performance embedded system for visual processing and control. We describe the actuation mechanism of the hand and the tendon-driven soft finger design with internally routed high-bandwidth flat-flex cables. For efficient on-board parallel processing of visual data from the cameras in each fingertip, we present a hybrid embedded architecture consisting of a field programmable logic array (FPGA) and a microcontroller that allows the realization of visual object segmentation based on convolutional neural networks. We evaluate the hand design by conducting durability experiments with one finger and quantify the grasp performance in terms of grasping force, speed and grasp success. The results show that the hand exhibits a grasp force of 31.8 ± 1.2 N and a mechanical durability of the finger of more than 15.000 closing cycles. Finally, we evaluate the accuracy of visual object segmentation during the different phases of the grasping process using five different objects. Hereby, an accuracy above 90% can be achieved.

Understanding spatial relations is a key element for natural human-robot interaction. Especially, a robot must be able to manipulate a given scene according to a human verbal command specifying desired spatial relations between objects. To endow robots with this ability, a suitable representation of spatial relations is necessary, which should be derivable from human demonstrations. We claim that polar coordinates can capture the underlying structure of spatial relations better than Cartesian coordinates and propose a parametric probability distribution defined in polar coordinates to represent spatial relations. We consider static spatial relations such as left of, behind, and near, as well as dynamic ones such as closer to and other side of, and take into account verbal modifiers such as roughly and a lot. We show that adequate distributions can be derived for various combinations of spatial relations and modifiers in a sample-efficient way using Maximum Likelihood Estimation, evaluate the effects of modifiers on the distribution parameters, and demonstrate our representation's usefulness in a pick-and-place task on a real robot.

In real world applications, robotic solutions remain impractical due to the challenges that arise in unknown and unstructured environments. To perform complex manipulation tasks in complex and cluttered situations, robots need to be able to identify the interaction possibilities with the scene, i.e. the affordances of the objects encountered. In unstructured environments with noisy perception, insufficient scene understanding and limited prior knowledge, this is a challenging task. In this work, we present an approach for grasping unknown objects in cluttered scenes with a humanoid robot in the context of a nuclear decommissioning task. Our approach combines the convenience and reliability of autonomous robot control with the precision and adaptability of teleoperation in a semi-autonomous selection of grasp affordances. Additionally, this allows exploiting the expert knowledge of an experienced human worker. To evaluate our approach, we conducted 75 real world experiments with more than 660 grasp executions on the humanoid robot ARMAR-6. The results demonstrate that high-level decisions made by the human operator, supported by autonomous robot control, contribute significantly to successful task execution.

Exoskeletons are considered to be a promising technology for assisting and augmenting human performance. A number of challenges related to design, intuitive control and interfaces to the human body must be addressed. In this paper, we approach the question of a minimal sensor setup for the realization of control strategies which take into account the actions currently performed by the user. To this end, we extend our previous work on online classifications of a human wearing a lower limb exoskeleton in two directions. First, we investigate the minimal number of sensors that should be attached to the exoskeleton to achieve a certain classification accuracy by investigating different sensor setups. We compare results of motion classification of 14 different daily activities such as walking forward and going upstairs using Hidden Markov Models. Second, we analyse the influence of different window sizes, as well as the classification performance of different motion types when training on multi- and single-subjects. Our results reveal that we can reduce our sensor setup significantly while achieving about the same classification performance.

Movement Primitives (MPs) are a promising way for representing robot motions in a flexible and adaptable manner. Due to the simple and compact form, they have been widely used in robotics. A major goal of the research activities on MPs is to learn models, which can adapt to changing task constraints, e.g. new motion targets. However, the adaptability of current MPs is limited to a small set of constraints due to their simple structures. It is indeed not a trivial task to maintain the simplicity of MPs representation and, at the same time, enhance their adaptability. In this paper, we discuss the adaptability of popular MPs such as Dynamic Movement Primitives (DMP) and Probabilistic Movement Primitives (ProMP) and propose a new simple but efficient formulation of MPs, the Via-points Movement Primitive (VMP), that can adapt to arbitrary via-points using a simple structured model that is based on the previous approaches but outperforms those in terms of extrapolation abilities.

Tactile sensors have been increasingly used to support rigid robot grippers in object grasping and manipulation. However, rigid grippers are often limited in their ability to handle compliant, delicate, or irregularly shaped objects. In recent years, grippers made from soft and flexible materials have become increasingly popular for certain manipulation tasks, e.g., grasping, due to their ability to conform to the object shape without the need for precise control. Although promising, such soft robot grippers currently suffer from the lack of available sensing modalities. In this work, we introduce a soft and stretchable sensing skin and incorporate it into the two fingers of a shape-memory actuated soft gripper. The onboard sensing skin includes a 9-axis inertial measurement unit (IMU) and five discrete pressure sensors per finger. We use this sensorized soft gripper to study grasp success and stability of over 2585 grasps with various objects using several machine learning methods. Our experiments show that LSTMs were the most accurate predictors of grasp success and stability, compared to SVMs, FFNNs, and ST-HMP. We also evaluated the effects on performance of each sensor's data, and the success rates for individual objects. The results show that the accelerometer data of the IMUs has the largest contribution to the overall grasp prediction, which we attribute to its ability to detect precise movements of the gripper during grasping.

Social robotic assistants have been widely studied and deployed as telepresence tools or caregivers. Evaluating their design and impact on the people interacting with them is of prime importance. In this research, we evaluate the usability and impact of ARMAR-6, an industrial robotic assistant for maintenance tasks. For this evaluation, we have used a modified System Usability Scale (SUS) to assess the general usability of the robotic system and the Godspeed questionnaire series for the subjective perception of the coworker. We have also recorded the subjects' gaze fixation patterns and analyzed how they differ when working with the robot compared to a human partner.

Despite impressive advances of humanoid robotics, the autonomous planning of whole-body loco-manipulation actions in unknown environments is still an open problem. In our previous work, we addressed two fundamental aspects related to this problem: 1) the autonomous detection of end-effector contact opportunities in unknown environments and 2) the goal-directed planning of multi-contact pose sequences, which can serve as the starting point for motion planning and control approaches of reduced complexity. Both problems suffer from the extensive amounts of possible solutions, particularly due to the complexity of humanoid robots and the multitude of available contact opportunities. In this paper, we propose a method for the planning of whole-body multi-contact tasks based on our previous work on vision-based detection of loco-manipulation affordances and whole-body multi-contact pose sequence planning. We demonstrate a combined approach for planning multi-contact pose sequences with a focus on the utilization of available end-effectors for stabilizing contacts with the environment during loco-manipulation tasks. The method is evaluated in simulation in multiple exemplary scenarios based on actual sensor data and the humanoid robot ARMAR-4.

We present a data-driven, bottom-up, deep learning approach to robotic grasping of unknown objects using Deep Convolutional Neural Networks (DCNNs). The approach uses depth images of the scene as its sole input for synthesis of a single-grasp solution during execution, adequately portraying the robot's visual perception during exploration of a scene. The training input consists of precomputed high-quality grasps, generated by analytical grasp planners, accompanied with rendered depth images of the training objects. In contrast to previous work on applying deep learning techniques to robotic grasping, our approach is able to handle full end-effector poses and therefore approach directions other than the view direction of the camera. Furthermore, the approach is not limited to a certain grasping setup (e. g. parallel jaw gripper) by design. We evaluate the method regarding its force-closure performance in simulation using the KIT and YCB object model datasets as well as a big data grasping database. We demonstrate the performance of our approach in qualitative grasping experiments on the humanoid robot ARMAR-III.

Intuitive exoskeleton control is fundamental since it contributes to improved user acceptance and wearability comfort. This requires the detection of user's motion intention and its incorporation into the exoskeleton control system. In this work, we propose a classification system based on Hidden Markov Models (HMMs), which facilitates the online classification of multi-modal sensor data acquired from a lower-limb exoskeleton based on previously defined motion patterns. For classification of these motion patterns at each time step, we consider the most recent sensor measurements by using a sliding window approach. We collected a training data set from a total number of 10 subjects performing 13 different motions with a passive exoskeleton equipped with 7 3D-force sensors and 3 inertial measurement units (IMUs). Our evaluation includes an analysis of the time needed for correct classification (latency), a validation for a training set containing all subjects and a leave-one-out validation to assess the generalization performance of the approach. The results indicate that our approach can classify motions of subjects included in the training set with an average accuracy of 92.80% and is able to achieve a generalization performance of 84.46%. With the selected parameters an average latency of 368.97 ms is achieved.

Tactile sensing of surface normals is essential for exploration of unknown objects. Many tactile sensors have been developed for contact measurement. However, few of these sensors provide surface orientation, and only up to a limited degree. This paper presents a novel contact and surface orientation sensor concept and its application for surface reconstruction of unknown objects. The sensor is comprised of an Inertial Measurement Unit (IMU) and a pressure sensor to accurately estimate the surface orientation in a wide range, while at the same time measuring contact force. We describe the developed sensor prototype and evaluate its performance regarding contact detection capability and normal estimation accuracy. We use this to reconstruct the surface of unknown objects using the humanoid robot ARMAR-III resulting in a mean reconstruction accuracy of 3.6 mm.

This work presents the concept of a robotic gripper designed for the disassembly of electromechanical devices that comprises several innovative ideas. Novel concepts include the ability to interchange built-in tools without the need to grasp them, the ability to reposition grasped objects in-hand, the capability of performing classic dual arm manipulation within the gripper and the utilization of classic industrial robotic arms kinematics within a robotic gripper. We analyze state of the art grippers and robotic hands designed for dexterous in-hand manipulation and extract common characteristics and weak points. The presented concept is obtained from the task requirements for disassembly of electromechanical devices and it is then evaluated for general purpose grasping, in-hand manipulation and operations with tools. We further present the CAD design for a first prototype.

The development and control of prosthetic hands is an active research area and recently progress in mechatronics, sensor integration and innovative control has been made. However, integration of different components into a prosthetic hand remains challenging due to space constraints, the requirements regarding holistic integration and the need for a user interface. In this paper, we present the KIT prosthetic hand, a novel five-finger 3D printed hand prosthesis, with its underactuated mechanism, sensors and embedded control system. The hand mechanics is based on the underactuated TUAT/Karlsruhe mechanism with two motors actuating 10 degrees of freedom. The mechanism has been realized in 3D printing technologies to facilitate a personalization of the prosthetic hand in terms of size and kinematic parameters. The prosthesis has been designed as a 50thpercentile male hand. It integrates an advanced embedded system as well as an RGB camera in the base of the palm and a colour display in the back of the hand. Experiments indicate a finger tip force of 7.48 N to 11.82 N, a hook grasp force of 120 N and a hand closing time of ~ 1.3 s.

Dynamic systems are a practical alternative to motion planning in executing robot actions. They are of particular interest in Learning from Demonstration, as here we aim to carry out actions in a certain fashion, without a model or in-depth knowledge about the world, which might be difficult to achieve with a planner. Using model-based dynamic systems in task space enables robots to flexibly reproduce demonstrated actions. Nevertheless, when dealing with mobile manipulators, we face the challenge of including the kinematic constraints of the robot in the action models. In this paper we propose to couple robot base and end-effector motions generated by arbitrary dynamical systems modulating the base velocity, while respecting the robots kinematic design. To this end we learn an approximation of the inverse reachability in closed form. In real-world robot experiments we demonstrate that we are able to maintain kinematically feasible trajectories in the presence of obstacles and in configurations differing profoundly from the training scene.

To increase the autonomy of humanoid robots, the visual perception must support the efficient collection and interpretation of visual scene cues by providing task-dependent information. Active vision systems allow to extend the observable workspace by employing active gaze control, i.e. by shifting the gaze to relevant areas in the scene. When moving the eyes, stabilization of the camera images is crucial for successful task execution. In this paper, we present an active vision system for task-oriented selection of view directions and gaze stabilization to enable a humanoid robot to robustly perform vision-based tasks. We investigate the interaction between a gaze stabilization controller and view planning to select the next best view direction based on saliency maps which encode task-relevant information. We demonstrate the performance of the systems in a real world scenario, in which a humanoid robot is performing vision-based grasping while moving, a task that would not be possible without the combination of view selection and gaze stabilization.

Robotic coverage path planning describes the problem of determining a configuration space trajectory for successively covering a specified workspace target area with the robot's end-effector. Performing coverage path planning for mobile robots further requires solving the problem of robot placement, i.e. determining of suitable robot base positions to perform the task. Finding an optimal solution is hard as both problems cannot be solved independently. Combined robot placement and coverage planning is particularly interesting if repositioning of the robot is costly or if simultaneous repositioning and end-effector motion is not desired. In this paper, we present a general approach for combined robot placement and coverage path planning that takes constraints like collision avoidance and static stability into account. In contrast to related approaches, we focus on mobile manipulation tasks that require a fixed placement for executing coverage trajectory segments. The approach is evaluated in two scenarios that exemplify the broad range of possible applications: The coverage of a building facade using a robotic manlift and the coverage of an industrial conveyer belt for maintenance tasks using the humanoid robot ARMAR-III.

An important question in imitation learning is how to generalize a learned motion to novel situations. The motion generalization depends on a set of features which can be represented as feature vectors spanning a feature space, called query space. The purpose of generalization is to find a mapping from this query space to the motion primitive space (MP space). In this paper, we address the problem of generalization of dynamic movement primitives (DMPs) to new queries by applying locally weighted regression (LWR) with radial basis functions (RBF). Since two DMPs differ only in their non-linear part, we transform the problem of DMP generalization to a regression analysis problem. We introduce a task-oriented regression algorithm with a cost function that takes task constraints into consideration and which relies on model switching to solve the problem of poor DMP generalization when using a single regression model for the entire query space. The evaluation shows that our algorithm outperforms related approaches in the literature in terms of generalization capabilities.

We address the question of how resource-aware concepts can be utilized in motion planning algorithms. Resource-awareness facilitate better resource allocation on global system level, e.g. when a humanoid robot needs to distribute and schedule a wide variety of concurrent algorithms. We present a motion planning approach that employs self-monitoring concepts in order to identify the difficulty of the planning problem. Resources are requested dynamically and adapted based on problem difficulty and current planning progress. We show how dynamic adaptation of resource allocation on algorithmic level can reduce the system workload as compared to static resource allocation while meeting Quality of Service (QoS) measures such as average workload or efficiency. We evaluate our approach both in several synthetic setups with varying difficulty and with the humanoid robot ARMAR-4.

We propose a formalism for the hierarchical representation of affordances. Starting with a perceived model of the environment consisting of geometric primitives like planes or cylinders, we define a hierarchical system for affordance extraction whose foundation are elementary power grasp affordances. Higher-level affordances, e.g. bimanual affordances, result from combining lower-level affordances with additional properties concerning the underlying geometric primitives of the scene. We model affordances as continuous certainty functions taking into account properties of the environmental elements and the perceiving robot's embodiment. The developed formalism is regarded as the basis for the description of whole-body affordances, i.e. affordances associated with whole-body actions. The proposed formalism was implemented and experimentally evaluated in multiple scenarios based on RGB-D camera data. The feasibility of the approach is demonstrated on a real robotic platform.

We present a novel approach for generating sequences of whole-body poses with multi-contacts for humanoid robots, which is inspired by techniques from natural language processing. To this end, we propose a probabilistic n-gram language model learned from observation of human locomotion tasks. Human motion data is automatically segmented according to detected contacts of the body with the environment to provide support, that is, support poses, which are further subdivided with regard to whole-body configuration. These poses are subsequently used to train a language model, whose words are the poses, and whose sentences represent sequences of poses. Then, we propose a planning algorithm that, given the constraints imposed by a task, finds the sequence of transitions with the highest probability according to our language model. We have applied our approach to 140 motion capture recordings of locomotion tasks that involve using one or both hands for support. The evaluation demonstrates that our approach is able to generate complex sets of pose transitions, and shows promising results regarding its application to more complex tasks.

Object shape information is essential for robot manipulation tasks, in particular for grasp planning and collision-free motion planning. But in general a complete object model is not available, in particular when dealing with unknown objects. We propose a method for completing shapes that are only partially known, which is a common situation when a robot perceives a new object only from one direction. Our approach is based on the assumption that most objects used in service robotic setups have symmetries. We determine and rate symmetry plane candidates to estimate the hidden parts of the object. By finding possible supporting planes based on its immediate neighborhood, the search space for symmetry planes is restricted, and the bottom part of the object is added. Gaps along the sides in the direction of the view axis are closed by linear interpolation. We evaluate our approach with real-world experiments using the YCB object and model set [1].

Dynamic movement primitives prove to be a useful and effective way to represent a movement of a given agent. However, the original DMP formulation does not take the interaction among multiple agents into the consideration. Thus, many researchers focus on the development of a coupling term for the underlying dynamical system and its associated learning strategies. The result is highly dependent on the quality of the learning methods. In this paper, we present a new way to formulate and realize interactive movement primitive in a leader-follower configuration, where the relationship between the follower and the leader is explicitly represented via the new formulation. This new formulation does not only simplify the learning process, but it also meets the requirements of several applications. We separately tested our new formulation in the context of the handover task and the wiping task. The results prove the flexibility and simplicity of the new formulation.

Large-scale human motion databases are key for research questions ranging from human motion analysis and synthesis, biomechanics of human motion, data-driven learning of motion primitives, and rehabilitation robotics to the design of humanoid robots and wearable robots such as exoskeletons. In this paper we present a large-scale database of whole-body human motion with methods and tools, which allows a unifying representation of captured human motion, and efficient search in the database, as well as the transfer of subject-specific motions to robots with different embodiments. To this end, captured subject-specific motion is normalized regarding the subject's height and weight by using a reference kinematics and dynamics model of the human body, the master motor map (MMM). In contrast with previous approaches and human motion databases, the motion data in our database consider not only the motions of the human subject but the position and motion of objects with which the subject is interacting as well. In addition to the description of the MMM reference model, we present procedures and techniques for the systematic recording, labeling, and organization of human motion capture data, object motions as well as the subject–object relations. To allow efficient search for certain motion types in the database, motion recordings are manually annotated with motion description tags organized in a tree structure. We demonstrate the transfer of human motion to humanoid robots and provide several examples of motion analysis using the database.

In this work we present a fast kinodynamic RRT-planner that uses dynamic nonprehensile actions to rearrange cluttered environments. In contrast to many previous works, the presented planner is not restricted to quasi-static interactions and monotonicity. Instead the results of dynamic robot actions are predicted using a black box physics model. Given a general set of primitive actions and a physics model, the planner randomly explores the configuration space of the environment to find a sequence of actions that transform the environment into some goal configuration. In contrast to a naive kinodynamic RRT-planner we show that we can exploit the physical fact that in an environment with friction any object eventually comes to rest. This allows a search on the configuration space rather than the state space, reducing the dimension of the search space by a factor of two without restricting us to non-dynamic interactions. We compare our algorithm against a naive kinodynamic RRT-planner and show that on a variety of environments we can achieve a higher planning success rate given a restricted time budget for planning.

We present a randomized kinodynamic planner that solves rearrangement planning problems. We embed a physics model into the planner to allow reasoning about interaction with objects in the environment. By carefully selecting this model, we are able to reduce our state and action space, gaining tractability in the search. The result is a planner capable of generating trajectories for full arm manipulation and simultaneous object interaction. We demonstrate the ability to solve more rearrangement by pushing tasks than existing primitive based solutions. Finally, we show the plans we generate are feasible for execution on a real robot.

This paper presents the development of a new type of robot capable of vertical and directional jumping. The robot uses soft silicone elastomer based pneumatic actuators as legs that accelerate the platform upwards by rapid pressurization. The robot is able to control and adjust the direction of the jumping by altering the timing patterns in which the individual legs are activated.

Exploiting interaction with the environment is a promising and powerful way to enhance stability of humanoid robots and robustness while executing locomotion and manipulation tasks. Recently some works have started to show advances in this direction considering humanoid locomotion with multi-contacts, but to be able to fully develop such abilities in a more autonomous way, we need to first understand and classify the variety of possible poses a humanoid robot can achieve to balance. To this end, we propose the adaptation of a successful idea widely used in the field of robot grasping to the field of humanoid balance with multi-contacts: a whole-body pose taxonomy classifying the set of whole-body robot configurations that use the environment to enhance stability. We have revised criteria of classification used to develop grasping taxonomies, focusing on structuring and simplifying the large number of possible poses the human body can adopt. We propose a taxonomy with 46 poses, containing three main categories, considering number and type of supports as well as possible transitions between poses. The taxonomy induces a classification of motion primitives based on the pose used for support, and a set of rules to store and generate new motions. We present preliminary results that apply known segmentation techniques to motion data from the KIT whole-body motion database. Using motion capture data with multi-contacts, we can identify support poses providing a segmentation that can distinguish between locomotion and manipulation parts of an action.

In this paper, we address the question of generative knowledge construction from sensorimotor experience, which is acquired by exploration. We show how actions and their effects on objects, together with perceptual representations of the objects, are used to build generative models which then can be used in internal simulation to predict the outcome of actions. Specifically, the paper presents an experiential cycle for learning association between object properties (softness and height) and action parameters for the wiping task and building generative models from sensorimotor experience resulting from wiping experiments. Object and action are linked to the observed effect to generate training data for learning a non-parametric continuous model using Support Vector Regression. In subsequent iterations, this model is grounded and used to make predictions on the expected effects for novel objects which can be used to constrain the parameter exploration. The cycle and skills have been implemented on the humanoid platform ARMAR-IIIb. Experiments with set of wiping objects differing in softness and height demonstrate efficient learning and adaptation behavior of action of wiping.

Overcoming the perceptual limitations of humanoid robots requires representations exploitable by highly integrable simulation, sensing, planning and acting components. Therefore, a novel active visual localization component for humanoid robots based on particle filtering in CAD environments is introduced. Specifically, two new components are presented: i) A vector-graphics prediction method employing hierarchical CAD environmental representations is presented. ii) A gaze attention method within the prediction-update cycle of the particle filter increases the available amount of visual features for localization while allowing adjustable task coupling. Finally, large and unobstructive ground-truth validation with the humanoid robot ARMAR-IIIb [1] in a made-for-humans environment shows the robustness, accuracy and performance of the proposed methods.

Autonomous robots often require domain knowledge to act intelligently in their environment. This is particularly true for robots that use automated planning techniques, which require symbolic representations of the operating environment and the robot's capabilities. However, the task of specifying domain knowledge by hand is tedious and prone to error. As a result, we aim to automate the process of acquiring general common sense knowledge of objects, relations, and actions, by extracting such information from large amounts of natural language text, written by humans for human readers. We present two methods for knowledge acquisition, requiring only limited human input, which focus on the inference of spatial relations from text. Although our approach is applicable to a range of domains and information, we only consider one type of knowledge here, namely object locations in a kitchen environment. As a proof of concept, we test our approach using an automated planner and show how the addition of common sense knowledge can improve the quality of the generated plans.

We present an approach for autonomous interactive object segmentation by a humanoid robot. The visual segmentation of unknown objects in a complex scene is an important prerequisite for e.g. object learning or grasping, but extremely difficult to achieve through passive observation only. Our approach uses the manipulative capabilities of humanoid robots to induce motion on the object and thus integrates the robots manipulation and sensing capabilities to segment previously unknown objects. We show that this is possible without any human guidance or pre-programmed knowledge, and that the resulting motion allows for reliable and complete segmentation of new objects in an unknown and cluttered environment. We extend our previous work, which was restricted to textured objects, by devising new methods for the generation of object hypotheses and the estimation of their motion after being pushed by the robot. These methods are mainly based on the analysis of motion of color annotated 3D points obtained from stereo vision, and allow the segmentation of textured as well as non-textured rigid objects. In order to evaluate the quality of the obtained segmentations, they are used to train a simple object recognizer. The approach has been implemented and tested on the humanoid robot ARMAR-III, and the experimental results confirm its applicability on a wide variety of objects even in highly cluttered scenes.

Successful and robust grasping for humanoid robots is still an ongoing research topic in robotics. Applying human-inspired grasping strategies does not only correspond with more natural looking motions but can also yield good results regarding task success when having to deal with uncertainty. This study investigates human high-level grasping strategies and how they tend to change for different objects when the uncertainty of object location or orientation increases in between two grasps. We are especially interested in potential gains for humanoid robots in a common household setting. By analyzing collected data from human subject grasp experiments with a set of typical objects found in people's homes, we get better insight into how humans handle uncertainty, as well as when and how they change their applied pre-grasp strategy. By adapting the by far most often observed change from a direct grasp attempt to a tapping strategy when dealing with high uncertainty, we can demonstrate a substantial increase of grasp success rate for our robot system with a Shadow Dexterous Hand mounted on a Motoman SDA10 robot while using less than two hand correction steps on average.

We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups based on whether they synthesize grasps for known, familiar, or unknown objects. This structure allows us to identify common object representations and perceptual processes that facilitate the employed data-driven grasp synthesis technique. In the case of known objects, we concentrate on the approaches that are based on object recognition and pose estimation. In the case of familiar objects, the techniques use some form of a similarity matching to a set of previously encountered objects. Finally, for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps. Our survey provides an overview of the different methodologies and discusses open problems in the area of robot grasping. We also draw a parallel to the classical approaches that rely on analytic formulations.

Having a representation of the capabilities of a robot is helpful when online queries, such as solving the inverse kinematics (IK) problem for grasping tasks, must be processed efficiently in the real world. When workspace representations, e.g. the reachability of an arm, are considered, additional quality information such as manipulability or self-distance can be employed to enrich the spatial data. In this work we present an approach of inverting such precomputed reachability representations in order to generate suitable robot base positions for grasping. Compared to existing works, our approach is able to generate a distribution in SE(2), the cross-space consisting of 2D position and 1D orientation, that describes potential robot base poses together with a quality index. We show how this distribution can be queried quickly in order to find oriented base poses from which a target grasping pose is reachable without collisions. The approach is evaluated in simulation using the humanoid robot ARMAR-III [1] and an extension is presented that allows to find suitable base poses for trajectory execution.

A major strength of humanoid robotics platforms consists in their potential to perform a wide range of manipulation tasks in human-centered environments thanks to their anthropomorphic design. Further, they offer active head-eye systems which allow to extend the observable workspace by employing active gaze control. In this work, we address the question where to look during manipulation tasks while exploiting these two key capabilities of humanoid robots. We present a solution to the gaze selection problem, which takes into account constraints derived from manipulation tasks. Thereby, three different subproblems are addressed: the representation of the acquired visual input, the calculation of saliency based on this representation, and the selection of the most suitable gaze direction. As representation of the visual input, a probabilistic environmental model is discussed, which allows to take into account the dynamic nature of manipulation tasks. At the core of the gaze selection mechanism, a novel saliency measure is proposed that includes accuracy requirements from the manipulation task in the saliency calculation. Finally, an iterative procedure based on spherical graphs is developed in order to decide for the best gaze direction. The feasibility of the approach is experimentally evaluated in the context of bimanual manipulation tasks on the humanoid robot ARMAR-III.

This paper presents a method for synthesizing motions of a humanoid robot that receives an object from a human, with focus on a natural object passing scenario where the human initiates the passing motion by moving an object towards the robot, which continuously adapts its motion to the observed human motion in real time. In this scenario, the robot not only has to recognize and adapt to the human action but also has to synthesize its motion quickly so that the human does not have to wait holding an object. We solve these issues by using a human motion database obtained from two persons performing the object passing task. The rationale behind this approach is that human performance of such a simple task is repeatable, and therefore the receiver (robot) motion can be synthesized by looking up the passer motion in a database. We demonstrate in simulation that the robot can start extending the arm at an appropriate timing and take hand configurations suitable for the object being passed. We also perform hardware experiments of object handing from a human to a robot.

In order to realize performance gain of a robot or an artificial arm, the end-effector which exhibits the same function as human beings and can respond to various objects and environment needs to be realized. Then, we developed the new hand which paid its attention to the structure of human being's hand which realize operation in human-like manipulation (called TUAT/Karlsruhe Humanoid Hand). Since this humanoid hand has the structure of adjusting grasp shape and grasp force automatically, it does not need a touch sensor and feedback control. It is designed for the humanoid robot which has to work autonomously or interactively in cooperation with humans and for an artificial arm for handicapped persons. The ideal end-effectors for such an artificial arm or a humanoid would be able to use the tools and objects that a person uses when working in the same environment. If this humanoid hand can operate the same tools, a machine and furniture, it may be possible to work under the same environment as human beings. As a result of adopting a new function of a palm and the thumb, the robot hand could do the operation which was impossible until now. The humanoid hand realized operations which hold a kitchen knife, grasping a fan, a stick, uses the scissors and uses chopsticks.

The framework of dynamic movement primitives allows the generation of discrete and periodic trajectories, which can be modulated in various aspects. We propose and evaluate a novel modulation approach that includes force feedback and thus allows physical interaction with objects and the environment. The proposed approach also enables the coupling of independently executed robotic trajectories, simplifying the execution of bimanual and tightly coupled cooperative tasks. We apply an iterative learning control algorithm to learn a coupling term, which is applied to the original trajectory in a feed-forward fashion. The coupling term modifies the trajectory in accordance to either the desired position or external force. The strengths of the approach are shown in bimanual or two-agent obstacle avoidance tasks, where no higher level cognitive reasoning or planning are required. Results of simulated and real-world experiments on the ARMAR-III humanoid robot in interaction and object lifting tasks, and on two KUKA LWR robots in a bimanual setting are presented.

Humanoid robots should be able to visually recognize objects and estimate their 6D pose in real environmental conditions with their limited sensor capabilities. In order to achieve these visual skills, it is necessary to establish an optimal visual transducer connecting the scene layout with the internal representations of objects and places. This visual transducer should capture the noiseless visual manifold of the scene with high-dynamic-range in an efficient manner. Our endeavor is to develop such a visual transducer using the widespread LDR cameras in humanoid robots. In our previous work, the noiseless acquisition of continuous images [1] and the improved radio-metric calibration [2] already enabled the humanoid robots to attain the desired visual manifold in terms of quality. However, since the radiance range of the scene can be very wide, the required amount of exposures to capture the visual manifold (robustly without radiance inconsistencies) turns impractically large in terms of scope, granularity and acquisition time. In this article, a method for estimating the minimal amount of exposures and their particular integration times is presented. This method integrates our previous work in order to synthesize HDR images with the minimal amount of exposures while ensuring the high quality of the resulting image. Conclusively, the minimal exposure set provides performance improvements without quality trade-off. Experimental evaluation is presented with the humanoid robots ARMAR-III a, b [3].

Generating robot motion trajectories instantaneously in the moment unforeseen sensor events happen is very essential for many real-world robot applications. Using a previous work on online trajectory generation as a basis, this paper proposes an alternative approach that also considers dynamic models. The former class of algorithms does not take into account dynamically changing acceleration capabilities based on maximum actuator forces/torques. This paper extends target velocity-based algorithms of the previous approach by taking into consideration the entire system dynamics when generating trajectories online within one control cycle (typically 1 ms or less). The extension includes the acceleration capabilities of a robot at every discrete time step assuming constant values for the maximum actuator forces/torques, thus allowing the generation of adaptive trajectory profiles during the motion of the robot. Several real-world experimental results using a seven-degree-of-freedom lightweight robot arm underline the relevance of this extension.

Most robotic vision algorithms are computationally intensive and operate on millions of pixels of real-time video sequences. But they offer a high degree of parallelism that can be exploited through parallel computing techniques like Invasive Computing. But the conventional way of multi-processing alone (with static resource allocation) is not sufficient enough to handle a scenario like robotic maneuver, where processing elements have to be shared between various applications and the computing requirements of such applications may not be known entirely at compile-time. Such static mapping schemes leads to inefficient utilization of resources. At the same time it is difficult to dynamically control and distribute resources among different applications running on a single chip, achieving high resource utilization under high-performance constraints. Invasive Computing obtains more importance under such circumstances, where it offers resource awareness to the application programs so that they can adapt themselves to the changing conditions, at run-time. In this paper we demonstrate the resource aware and self-organizing behavior of invasive applications using three widely used applications from the area of robotic vision - Optical Flow, Object Recognition and Disparity Map Computation. The applications can dynamically acquire and release hardware resources, considering the level of parallelism available in the algorithm and time-varying load.

The ability to grasp unknown objects is an important skill for personal robots, which has been addressed by many present and past research projects, but still remains an open problem. A crucial aspect of grasping is choosing an appropriate grasp configuration, i.e. the 6d pose of the hand relative to the object and its finger configuration. Finding feasible grasp configurations for novel objects, however, is challenging because of the huge variety in shape and size of these objects. Moreover, possible configurations also depend on the specific kinematics of the robotic arm and hand in use. In this paper, we introduce a new grasp selection algorithm able to find object grasp poses based on previously demonstrated grasps. Assuming that objects with similar shapes can be grasped in a similar way, we associate to each demonstrated grasp a grasp template. The template is a local shape descriptor for a possible grasp pose and is constructed using 3d information from depth sensors. For each new object to grasp, the algorithm then finds the best grasp candidate in the library of templates. The grasp selection is also able to improve over time using the information of previous grasp attempts to adapt the ranking of the templates. We tested the algorithm on two different platforms, the Willow Garage PR2 and the Barrett WAM arm which have very different hands. Our results show that the algorithm is able to find good grasp configurations for a large set of objects from a relatively small set of demonstrations, and does indeed improve its performance over time.

Planning motion for humanoid robots requires obeying simultaneous constraints on balance, collision-avoidance, and end-effector pose, among others. Several algorithms are able to generate configurations that satisfy these constraints given a good initial guess, i.e. a configuration which is already close to satisfying the constraints. However, when selecting goals for a planner a close initial guess is rarely available. Methods that attempt to satisfy all constraints through direct projection from a distant initial guess often fail due to opposing gradients for the various constraints, joint-limits, or singularities. We approach the problem of generating a constrained goal by searching for a configuration in the intersection of all constraint manifolds in configuration space (C-space). Starting with an initial guess, our algorithm, Constellation, builds a graph in C-space whose nodes are configurations that satisfy one or more constraints and whose cycles determine where the algorithm explores next. We compare the performance of our approach to direct projection and a previously-proposed cyclic projection method on reaching tasks for a humanoid robot with 33 DOF. We find that Constellation performs the best in terms of the number of solved queries across a wide range of problem difficulty. However, this success comes at higher computational cost.

The previously presented Kinematic Bézier Maps (KBM) are a machine learning algorithm that has been tailored to efficiently learn the kinematics of redundant robots. This algorithm relies upon a representation based on projective geometry that uses a special set of polynomial functions borrowed from the field of Computer Aided Geometric Design (CAGD). So far, it has only been possible to learn a model of the forward kinematics function. In this paper, we show how the KBM algorithm can be modified to learn the robot's equation of motion and, hence, its inverse dynamic model. Results from experiments with a simulated serial robot manipulator are presented that clearly show the advantages of our approach compared to general function approximation methods.

The calibration of serial manipulators with high numbers of degrees of freedom by means of machine learning is a complex and time-consuming task. With the help of a simple strategy, this complexity can be drastically reduced and the speed of the learning procedure can be increased. When the robot is virtually divided into shorter kinematic chains, these subchains can be learned separately and hence much more efficiently than the complete kinematics. Such decompositions, however, require either the possibility to capture the poses of all end effectors of all subchains at the same time, or they are limited to robots that fulfill special constraints. In this paper, an alternative decomposition is presented that does not suffer from these limitations. An offline training algorithm is provided in which the composite subchains are learned sequentially with dedicated movements. A second training scheme is provided to train composite chains simultaneously and online. Both schemes can be used together with many machine learning algorithms. In the simulations, an algorithm using parameterized self-organizing maps modified for online learning and Gaussian mixture models (GMMs) were chosen to show the correctness of the approach. The experimental results show that, using a twofold decomposition, the number of samples required to reach a given precision is reduced to twice the square root of the original number.

The kinematics of a robot with many degrees of freedom is a very complex function. Learning this function for a large workspace with a good precision requires a huge number of training samples, i.e., robot movements. In this paper, we introduce the Kinematic Bézier Map (KB-Map), a parameterizable model without the generality of other systems but whose structure readily incorporates some of the geometric constraints of a kinematic function. In this way, the number of training samples required is drastically reduced. Moreover, the simplicity of the model reduces learning to solving a linear least squares problem. Systematic experiments have been carried out showing the excellent interpolation and extrapolation capabilities of KB-Maps and their relatively low sensitivity to noise.

Image-based 6-DoF pose estimation of arbitrarily shaped 3D objects based on their shape is a rarely studied problem. Most existing image-based methods for pose estimation either exploit textural information in form of local features or, if shape-based, rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but in practice assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on its relative pose to the observing camera. One way to reduce the solution space is to exploit temporal information, i.e. perform tracking. Again, existing model-based tracking approaches rely on relatively simple object geometries. In this paper, we propose a particle filter based tracking approach that can deal with arbitrary shapes and arbitrary or even no texture, i.e. it offers a general solution to the rigid object tracking problem. As our approach can deal with occlusions, it is in particular of interest in the context of goal-directed imitation learning involving the observation of object manipulations. Results of simulation experiments as well as real-world experiments with different object types prove the practical applicability of our approach.

In this paper, we present a grasp representation in task space exploiting position information of the fingertips. We propose a new way for grasp representation in the task space, which provides a suitable basis for grasp imitation learning. Inspired by neuroscientific findings, finger movement synergies in the task space together with fingertip positions are used to derive a parametric low-dimensional grasp representation. Taking into account correlating finger movements, we describe grasps using a system of virtual springs to connect the fingers, where different grasp types are defined by parameterizing the spring constants. Based on such continuous parameterization, all instantiation of grasp types and all hand preshapes during a grasping action (reach, preshape, enclose, open) can be represented. We present experimental results, in which the spring constants are merely estimated from fingertip motion tracking using a stereo camera setup of a humanoid robot. The results show that the generated grasps based on the proposed representation are similar to the observed grasps.

Humanoid robots should be able to grasp and handle objects in the environment, even if the objects are seen for the first time. A plausible solution to this problem is to categorize these objects into existing classes with associated actions and functional knowledge. So far, efforts on visual object categorization using humanoid robots have either been focused on appearance-based methods or have been restricted to object recognition without generalization capabilities. In this work, a shape model-based approach using stereo vision and machine learning for object categorization is introduced. The state-of-the-art features for shape matching and shape retrieval were evaluated and selectively transfered into the visual categorization. Visual sensing from different vantage points allows the reconstruction of 3D mesh models of the objects found in the scene by exploiting knowledge about the environment for model-based segmentation and registration. These reconstructed 3D mesh models were used for shape feature extraction for categorization and provide sufficient information for grasping and manipulation. Finally, the visual categorization was successfully performed with a variety of features and classifiers allowing proper categorization of unknown objects even when object appearance and shape substantially differ from the training set. Experimental evaluation with the humanoid robot ARMAR-IIIa is presented.

In this paper parameter-free concepts for exact motion planning are investigated. With the proposed RDT+approach the collision detection parameters of a Rapidly exploring Dense Tree (RDT) are automatically adjusted until an exact solution can be found. For efficient planning discrete collision detection routines are used within the RDT planner and by verifying the results with exact collision detection methods, the RDT+ concept allows to compute motions that are guaranteed collision-free. We show the probabilistic completeness of the proposed planner and present an extension for handling narrow passages. The algorithms are evaluated in different experiments, including narrow passages and high-dimensional planning problems, that are solved in simulation and on the humanoid robot ARMAR-III.

We introduce an approach for enabling sampling-based planners to compute motions with humanlike appearance. The proposed method is based on a space of blendable example motions collected by motion capture. This space is explored by a sampling-based planner that is able to produce motions around obstacles while keeping solutions similar to the original examples. The results therefore largely maintain the humanlike characteristics observed in the example motions. The method is applied to generic upper-body actions and is complemented by a locomotion planner that searches for suitable body placements for executing upper-body actions successfully. As a result, our overall multi-modal planning method is able to automatically coordinate whole-body motions for action execution among obstacles, and the produced motions remain similar to example motions given as input to the system.

In this work, we present a new software environment for the comparative evaluation of algorithms for grasping and dexterous manipulation. The key aspect in its development is to provide a tool that allows the reproduction of well-defined experiments in real-life scenarios in every laboratory and, hence, benchmarks that pave the way for objective comparison and competition in the field of grasping. In order to achieve this, experiments are performed on a sound open-source software platform with an extendable structure in order to be able to include a wider range of benchmarks defined by robotics researchers. The environment is integrated into the OpenGRASP toolkit that is built upon the OpenRAVE project and includes grasp-specific extensions and a tool for the creation/integration of new robot models. Currently, benchmarks for grasp and motion planningare included as case studies, as well as a library of domestic everyday objects models, and a real-life scenario that features a humanoid robot acting in a kitchen.

In this paper we propose the basic concept of a tele-presence system for two (or more) anthropomorphic robots located in remote locations. As one robot interacts with a user, it acquires knowledge about the user's behavior and transfers this knowledge to the network. The robot in the remote location accesses this knowledge and according to this information emulates the behavior of the remote user when interacting with its partner. The behavioral patterns are grouped into macro-behavior units (maBUs) and mirco-behavior units (miBUs). maBUs carry information that is specific to a person-to-person communication. miBUs are commonly used behavior patterns within a certain cultural environment (e.g. handshake, bow etc). maBUs are usually chains of miBUs. miBUs are chains of expression actions (e.g. gesture, facial expression etc.). The idea behind this is, that human communication contains several levels or layers of information exchange. This to be emulated by implementing the concept of miBUs and maBUs. We present a preliminary application of this concept to a musical context. The rhythmic motion of a drum-stick during the performance of a drum rhythm by a musician is recorded by a inertial measurements unit and transmitted between two far distance locations (Waseda University in Japan and Karlsruhe Institute of Technology in Germany) using three different transmission methods: direct raw data transmission, miBU based transmission and maBU-based transmission. We present experimental results that show quantitative data to evaluate the suitability of our approach, with the overall goal to implement a telepresence system of larger scale.

In this work, we present an integrated planner for collision-free single and dual arm grasping motions. The proposed Grasp-RRT planner combines the three main tasks needed for grasping an object: finding a feasible grasp, solving the inverse kinematics and searching a collision-free trajectory that brings the hand to the grasping pose. Therefore, RRT-based algorithms are used to build a tree of reachable and collision-free configurations. During RRT-generation, potential grasping positions are generated and approach movements toward them are computed. The quality of reachable grasping poses is scored with an online grasp quality measurement module which is based on the computation of applied forces in order to diminish the net torque.We also present an extension to a dual arm planner which generates bimanual grasps together with corresponding dual arm grasping motions. The algorithms are evaluated with different setups in simulation and on the humanoid robot ARMAR-III.

The autonomous acquisition of object representations which allow recognition, localization and grasping of objects in the environment is a challenging task, which has shown to be difficult. In this paper, we present a systems for autonomous acquisition of visual object representations, which endows a humanoid robot with the ability to enrich its internal object representation and allows the realization of complex visual tasks. More precisely, we present techniques for segmentation and modeling of objects held in the five-fingered robot hand. Multiple object views are generated by rotating the held objects in the robot's field of view. The acquired object representations are evaluated in the context of visual search and object recognition tasks in cluttered environments. Experimental results show successful implementation of the complete cycle from object exploration to object recognition on a humanoid robot.

Typical tasks of future service robots involve grasping and manipulating a large variety of objects differing in size and shape. Generating stable grasps on 3D objects is considered to be a hard problem, since many parameters such as hand kinematics, object geometry, material properties and forces have to be taken into account. This results in a high-dimensional space of possible grasps that cannot be searched exhaustively. We believe that the key to find stable grasps in an efficient manner is to use a special representation of the object geometry that can be easily analyzed. In this paper, we present a novel grasp planning method that evaluates local symmetry properties of objects to generate only candidate grasps that are likely to be of good quality. We achieve this by computing the medial axis which represents a 3D object as a union of balls. We analyze the symmetry information contained in the medial axis and use a set of heuristics to generate geometrically and kinematically reasonable candidate grasps. These candidate grasps are tested for force-closure. We present the algorithm and show experimental results on various object models using an anthropomorphic hand of a humanoid robot in simulation.

Acquisition of new sensorimotor knowledge by imitation is a promising paradigm for robot learning. To be effective, action learning should not be limited to direct replication of movements obtained during training but must also enable the generation of actions in situations a robot has never encountered before. This paper describes a methodology that enables the generalization of the available sensorimotor knowledge. New actions are synthesized by the application of statistical methods, where the goal and other characteristics of an action are utilized as queries to create a suitable control policy, taking into account the current state of the world. Nonlinear dynamic systems are employed as a motor representation. The proposed approach enables the generation of a wide range of policies without requiring an expert to modify the underlying representations to account for different task-specific features and perceptual feedback. The paper also demonstrates that the proposed methodology can be integrated with an active vision system of a humanoid robot. 3-D vision data are used to provide query points for statistical generalization. While 3-D vision on humanoid robots with complex oculomotor systems is often difficult due to the modeling uncertainties, we show that these uncertainties can be accounted for by the proposed approach.

We provide a general approach for learning robotic motor skills from human demonstration. To represent an observed movement, a non-linear differential equation is learned such that it reproduces this movement. Based on this representation, we build a library of movements by labeling each recorded movement according to task and context (e.g., grasping, placing, and releasing). Our differential equation is formulated such that generalization can be achieved simply by adapting a start and a goal parameter in the equation to the desired position values of a movement. For object manipulation, we present how our framework extends to the control of gripper orientation and finger position. The feasibility of our approach is demonstrated in simulation as well as on the Sarcos dextrous robot arm. The robot learned a pick-and-place operation and a water-serving task and could generalize these tasks to novel situations.

Visual search is a common daily human activity and a prerequisite to the interaction with objects encountered in cluttered environments. Humanoid robots that are supposed to take part in human daily life should possess similar capabilities in terms of representing, attending to and recalling objects of interest in order to ensure robust perception in human-centered environments. In this paper, we present necessary processes, memories and representations which allow to identify and store locations of objects, encountered from different angles of view, in a visual search task. In particular, we introduce the so-called Feature Ego-Sphere (FES) as the scene memory for a humanoid robot. Experiments comprising different visual search tasks have been carried out on an active humanoid head equipped with perspective and foveal stereo camera systems. The scene is analyzed actively using both camera systems in order to find instances of searched objects in a consistent and persistent manner.

The problem of accurate 6-DoF pose estimation of 3D objects based on their shape has so far been solved only for specific object geometries. Edge-based recognition and tracking methods rely on the extraction of straight line segments or other primitives. Straight-forward extensions of 2D approaches are potentially more general, but assume a limited range of possible view angles. The general problem is that a 3D object can potentially produce completely different 2D projections depending on the view angle. One way to tackle this problem is to use canonical views. However, accurate shape-based 6-DoF pose estimation requires more information than matching of canonical views can provide. In this paper, we present a novel approach to 6-DoF pose estimation of single-colored objects based on their shape. Our approach combines stereo triangulation with matching against a high-resolution view set of the object, each view having associated orientation information. The errors that arise from separating the position and orientation computation in first place are corrected by a subsequent correction procedure based on online 3D model projection. The proposed approach can estimate the pose of a single object within 20 ms using conventional hardware.

In the recent past, the recognition and localization of objects based on local point features has become a widely accepted and utilized method. Among the most popular features are currently the SIFT features, the more recent SURF features, and region-based features such as the MSER. For time-critical application of object recognition and localization systems operating on such features, the SIFT features are too slow (500-600 ms for images of size 640×480 on a 3 GHz CPU). The faster SURF achieve a computation time of 150-240 ms, which is still too slow for active tracking of objects or visual servoing applications. In this paper, we present a combination of the Harris corner detector and the SIFT descriptor, which computes features with a high repeatability and very good matching properties within approx. 20 ms. While just computing the SIFT descriptors for computed Harris interest points would lead to an approach that is not scale-invariant, we will show how scale-invariance can be achieved without a time-consuming scale space analysis. Furthermore, we will present results of successful application of the proposed features within our system for recognition and localization of textured objects. An extensive experimental evaluation proves the practical applicability of our approach.

In this paper, we present efficient solutions for planning motions of dual-arm manipulation and re-grasping tasks. Motion planning for such tasks on humanoid robots with a high number of degrees of freedom (DoF) requires computationally efficient approaches to determine the robot's full joint configuration at a given grasping position, i.e. solving the Inverse Kinematics (IK) problem for one or both hands of the robot. In this context, we investigate solving the inverse kinematics problem and motion planning for dual-arm manipulation and re-grasping tasks by combining a gradient-descent approach in the robot's pre-computed reachability space with random sampling of free parameters. This strategy provides feasible IK solutions at a low computation cost without resorting to iterative methods which could be trapped by joint-limits. We apply this strategy to dual-arm motion planning tasks in which the robot is holding an object with one hand in order to generate whole-body robot configurations suitable for grasping the object with both hands. In addition, we present two probabilistically complete RRT-based motion planning algorithms (J+-RRT and IK-RRT) that interleave the search for an IK solution with the search for a collision-free trajectory and the extension of these planners to solving re-grasping problems. The capabilities of combining IK methods and planners are shown both in simulation and on the humanoid robot ARMAR-III performing dual-arm tasks in a kitchen environment.

In this paper, a geometric approach for global self-localization based on a world-model and active stereo vision is introduced. The method uses class specific object recognition algorithms to obtain the location of entities within the surroundings. The perceived entities in recognition trials are simultaneously filtered and fused to provide a robust set of class features. These classified perceptions which simultaneously satisfy geometric and topological constraints are employed for pruning purposes upon the world-model generating the location hypotheses set. Finally, the hypotheses are validated and disambiguated by applying visual recognition algorithms to selected entities of the world-model. The proposed approach has been successfully used with a humanoid robot.

In this paper we study object recognition on a humanoid robotic head. The head is equipped with a stereo vision system with two cameras in each eye, where the cameras have lenses with different view angles. Such a system models the foveated structure of a human eye. To facilitate the pursuit of moving objects, we provide mathematical analysis that enables the robot to guide the narrow-view cameras toward the object of interest based on information extracted from the wider views. Images acquired by narrow-view cameras, which produce object images at higher resolutions, are used for recognition. The proposed recognition approach is view-based and is built around a classifier using nonlinear multi-class support vector machines with a special kernel function. We show experimentally that the increased resolution leads to higher recognition rates.

Daily life objects reveal natural similarities, which cannot be resolved with the perception of a single view. In this paper, we present an approach for object separation using active methods and multi-view object representations. By actively rotating an object, the coherence between controlled path, inner models, and percept is observed and used to reject implausible object hypotheses. Using the resulting object hypotheses, pose and object correspondence are determined. The proposed approach allows for the separation of different object candidates, which have similar views to the current percept. With the benefit of active methods the perceptual task can be solved using even coarse features, which facilitates a compact multi-view object representation. Furthermore, the approach is independent from a specific visual feature descriptor and thus suitable for multi-modal object recognition.

Motion planning for robots with many degrees of freedom (DoF) is a generally unsolved problem in the robotics context. In this work an approach for trajectory planning is presented, which takes account of the different kinematic parts of a humanoid robot. Since not all joints of the robot are important for different planning phases, the RRT-based planner is able to adapt the number of DoF on the fly to improve the performance and the quality of the results. The runtime of the approach is evaluated in comparison to a standard RRT planner. Futhermore several extensions to the algorithm are investigated.

Summary form only given. The development and emergence of cognition relies on artificial embodiments having complex and rich perceptual and motor capabilities. The impressive advance of research and development in robotics over the past years has led to the development of humanoid robots that are rich in sensory and motor capabilities and hence provide a suitable framework for studying cognition. Currently, the different disciplines related to the development of cognitive humanoids have usually been explored independently, leading to significant results within each discipline. However, the big challenge is how different pieces of results fit together to achieve complete processing models and an integrative system architecture, and how to evaluate results at system level rather than focusing on the performance of component algorithms.

In this paper, we present a framework for perception, visualization, reproduction and recognition of human motion. On the perception side, various human motion capture systems exist, all of them having in common to calculate a sequence of configuration vectors for the human model in the core of the system. These human models may be 2D or 3D kinematic models, or on a lower level, 2D or 3D positions of markers. However, for appropriate visualization in terms of a 3D animation, and for reproduction on an actual robot, the acquired motion must be mapped to the target 3D kinematic model. On the understanding side, various action and activity recognition systems exist, which assume input of different kinds. However, given human motion capture data in terms of a high-dimensional 3D kinematic model, it is possible to transform the configurations into the appropriate representation which is specific to the recognition module. We will propose a complete architecture, allowing the replacement of any perception, visualization, reproduction module, or target platform. In the core of our architecture, we define a reference 3D kinematic model, which we intend to become a common standard in the robotics community, to allow sharing different software modules and having common benchmarks.

In this paper, we present an image-based markerless human motion capture system, intended for humanoid robot systems. The restrictions set by this ambitious goal are numerous. The input of the system is a sequence of stereo image pairs only, captured by cameras positioned at approximately eye distance. No artificial markers can be used to simplify the estimation problem. Furthermore, the complexity of all algorithms incorporated must be suitable for real-time application, which is maybe the biggest problem when considering the high dimensionality of the search space. Finally, the system must not depend on a static camera setup and has to find the initial configuration automatically. We present a system, which tackles these problems by combining multiple cues within a particle filter framework, allowing the system to recover from wrong estimations in a natural way. We make extensive use of the benefit of having a calibrated stereo setup. To reduce search space implicitly, we use the 3D positions of the hands and the head, computed by a separate hand and head tracker using a linear motion model for each entity to be tracked. With stereo input image sequences at a resolution of 320 times 240 pixels, the processing rate of our system is 15 Hz on a 3 GHz CPU. Experimental results documenting the performance of our system are available in form of several videos.

This paper presents the resolve spatial constraints (RSC) algorithm for manipulation planning in a domain with movable obstacles. Empirically we show that our algorithm quickly generates plans for simulated articulated robots in a highly nonlinear search space of exponential dimension. RSC is a reverse-time search that samples future robot actions and constrains the space of prior object displacements. To optimize the efficiency of RSC, we identify methods for sampling object surfaces and generating connecting paths between grasps and placements. In addition to experimental analysis of RSC, this paper looks into object placements and task-space motion constraints among other unique features of the three dimensional manipulation planning domain.

Robust vision-based grasping is still a hard problem for humanoid robot systems. When being restricted to using the camera system built-in into the robot's head for object localization, the scenarios get often very simplified in order to allow the robot to grasp autonomously. Within the computer vision community, many object recognition and localization systems exist, but in general, they are not tailored to the application on a humanoid robot. In particular, accurate 6D object localization in the camera coordinate system with respect to a 3D rigid model is crucial for a general framework for grasping. While many approaches try to avoid the use of stereo calibration, we will present a system that makes explicit use of the stereo camera system in order to achieve maximum depth accuracy. Our system can deal with textured objects as well as objects that can be segmented globally and are defined by their shape. Thus, it covers the cases of objects with complex texture and complex shape. Our work is directly linked to a grasping framework being implemented on the humanoid robot ARM AR and serves as its perception module for various grasping and manipulation experiments in a kitchen scenario.

Motion planning for humanoid robotic systems with many degrees of freedom is an important and still generally unsolved problem. To give the robot the ability of acting and navigating in complex environments, the motion planner has to find collision-free paths in a robust manner. The runtime of a planning algorithm is critical, since complex tasks require several planning steps where the collision detection and avoidance should be accomplished in reasonable time. In this paper we present an extension of standard sampling-based techniques using Rapidly Exploring Random Trees (RRT). We extend the free-bubble path validation algorithm from Quinlan, which can be used to guarantee the collision-free status of a C-space path between two samples. By using enlarged robot models it is possible to avoid costly distance calculations and therefore to speed up the planning process. We also present a combined approach based on lazy collision checking that brings together the advantages of fast sampling-based and exact path-validated algorithms. The proposed algorithms have been evaluated by experiments on a humanoid robot in a kitchen environment and by a comparison to a validation based on Quinlan's free bubbles approach.

We propose a novel solution to the problem of inverse kinematics for redundant robotic manipulators for the purposes of goal selection for path planning. We unify the calculation of the goal configuration with searching for a path in order to avoid the uncertainties inherent to selecting goal configurations which may be unreachable because they currently lie in components of the free configuration space disconnected from the initial configuration. We adopt workspace heuristic functions that implicitly define goal regions of the configuration space and guide the extension of rapidly-exploring random trees (RRTs), which are used to search for these regions. The algorithm has successfully been used to efficiently plan reaching and grasping motions for a humanoid robot equipped with redundant manipulator arms

A general solution for image-based object recognition and localization is still a goal far away. Therefore, the only way to tackle the problem is to apply the suitable approach for each specific problem. The most common techniques can be classified into global appearance-based, model-based, or histogram-based approaches, and approaches based on local features. In this paper, we concentrate on recognition and full 6D localization of solid colored objects of any geometry for real-time application on a humanoid robot system. State-of-the-art model-based methods can only deal with object geometries which can be broken down into 3D lines and planes, and thus can be efficiently projected into the image plane, which is not the case for most objects in a realistic scenario. In contrast, appearance-based methods have the power to be applicable for any object geometry, but are rarely combined with full 6D localization of objects, which is required for any realistic application in the context of grasping with a humanoid robot. We present a system which combines the benefits of global appearance-based and model-based approaches, resulting in a system which can acquire object representations automatically given its 3D model, and can recognize and localize solid-colored objects in 6D in an arbitrary scene in real-time

In this paper we present a framework for grasp planning with a humanoid robot arm and a five-fingered hand. The aim is to provide the humanoid robot with the ability of grasping objects that appear in a kitchen environment. Our approach is based on the use of an object model database that contains the description of all the objects that can appear in the robot workspace. This database is completed with two modules that make use of this object representation: an exhaustive offline grasp analysis system and a real-time stereo vision system. The offline grasp analysis system determines the best grasp for the objects by employing a simulation system, together with CAD models of the objects and the five-fingered hand. The results of this analysis are added to the object database using a description suited to the requirements of the grasp execution modules. A stereo camera system is used for a real-time object localization using a combination of appearance-based and model-based methods. The different components are integrated in a controller architecture to achieve manipulation task goals for the humanoid robot

In this paper we present a modular and distributed control architecture in order to achieve natural interaction and mobile manipulation task goals for a humanoid robot. We propose a hierarchically organized architecture with three levels and introduce the mapping of the functional features in this architecture into hardware and software modules. We also describe different functional features which have been realized and integrated into the whole control architecture.

This paper deals with easy programming methods of dual-arm manipulation tasks for humanoid robots. Hereby a programming by demonstration system is used in order to observe, learn and generalize tasks performed by humans. A classification for dual-arm manipulations is introduced, enabling a segmentation of tasks into adequate subtasks. Further it is shown how the generated programs are mapped on and executed by a humanoid robot.

Humanoid robotics is a new challenging field. To cooperate with human beings, humanoid robots not only have to feature human-like form and structure but, more importantly, they must possess human-like characteristics regarding motion, communication and intelligence. In this paper, we propose an algorithm for solving the inverse kinematics problem associated with the redundant robot arm of the humanoid robot ARMAR. The formulation of the problem is based on the decomposition of the workspace of the arm and on the analytical description of the redundancy of the arm. The solution obtained is characterized by its accuracy and low cost of computation. The algorithm is enhanced in order to generate human-like manipulation motions from object trajectories.

The increasing demand for robotic applications in dynamic unstructured environments is motivating the need for dextrous end-effectors which can cope with the wide variety of tasks and objects encountered in these environments. The human hand is a very complex grasping tool that can handle objects of different sizes and shapes. Many research activities have been carried out to develop artificial robot hands with capabilities similar to the human hand. In this paper the mechanism and design of a new humanoid-type hand (called TUAT/Karlsruhe Humanoid Hand) with human-like manipulation abilities is discussed. The new hand is designed for the humanoid robot ARMAR which has to work autonomously or interactively in cooperation with humans and for an artificial lightweight arm for handicapped persons. The arm is developed as close as possible to the human arm and is driven by spherical ultrasonic motors. The ideal end-effector for such an artificial arm or a humanoid would be able to use the tools and objects that a person uses when working in the same environment. Therefore a new hand is designed for anatomical consistency with the human hand. This includes the number of fingers and the placement and motion of the thumb, the proportions of the link lengths and the shape of the palm. It can also perform most part of human grasping types. The TUAT/Karlsruhe Humanoid Hand possesses 20 DOF and is driven by one actuator which can be placed into or around the hand.

Service robots which should perform human-like operations will penetrate into a great number of applications in the future. Requirements for this is high flexibility, autonomy and the ability to adapt to new situations. The paper describes a design concept and a prototype implementation of an autonomous mobile humanoid service robot, which should mainly support people in their daily life as a personal or an assistance robot. The state of the research is that the general concept is developed and two anthropomorphic arms are built up. In the article the sensor system and the control architecture of the anthropomorphic robot are described. To evaluate the performance and motion abilities of the anthropomorphic arm the human arm kinematics and properties are discussed.