Recent advances in deep neural networks have opened up new possibilities for visuomotor robot learning. In the context of human-robot or robot-robot collaboration, such networks can be trained to predict future poses and this information can be used to improve the dynamics of cooperative tasks. This is important, both in terms of realizing various cooperative behaviors, and for ensuring safety. In this article, we propose a recurrent neural architecture, capable of transforming variable-length input motion videos into a set of parameters describing a robot trajectory, where predictions can be made after receiving only a few frames. A simulation environment is utilized to expand the training database and to improve generalization capability of the network. The resulting architecture demonstrates good accuracy when predicting handover trajectories, with models trained on synthetic and real data showing better performance than when trained on real or simulated data only. The computed trajectories enable the execution of handover tasks with uncalibrated robots, which was verified in an experiment with two real robots.

This paper presents a study on the use of the Talos humanoid robot for performing assistive sit-to-stand or stand-to-sit tasks. In such tasks, the human exerts a large amount of force (100–200 N) within a very short time (2–8 s), posing significant challenges in terms of human unpredictability and robot stability control. To address these challenges, we propose an approach for finding a spatial reference for the robot, which allows the robot to move according to the force exerted by the human and control its stability during the task. Specifically, we focus on the problem of finding a 1D manifold for the robot, while assuming a simple controller to guide its movement on this manifold. To achieve this, we use a functional representation to parameterize the manifold and solve an optimization problem that takes into account the robot's stability and the unpredictability of human behavior. We demonstrate the effectiveness of our approach through simulations and experiments with the Talos robot, showing robustness and adaptability.

We propose a new approach to programming by the demonstration of finishing operations. Such operations can be carried out by industrial robots in multiple ways because an industrial robot is typically functionally redundant with respect to a finishing task. In the proposed system, a human expert demonstrates a finishing operation, and the demonstrated motion is recorded in the Cartesian space. The robot’s kinematic model is augmented with a virtual mechanism, which is defined according to the applied finishing tool. This way, the kinematic model is expanded with additional degrees of freedom that can be exploited to compute the optimal joint space motion of the robot without altering the essential aspects of the Cartesian space task execution as demonstrated by the human expert. Finishing operations, such as polishing and grinding, occur in contact with the treated workpiece. Since information about the contact point position is needed to control the robot during the operation, we have developed a novel approach for accurate estimation of contact points using the measured forces and torques. Finally, we applied iterative learning control to refine the demonstrated operations and compensate for inaccurate calibration and different dynamics of the robot and human demonstrator. The proposed method was verified on real robots and real polishing and grinding tasks.Note to Practitioners—This work was motivated by the need for automation of finishing operations, such as polishing and grinding, on contemporary industrial robots. Existing approaches are both too complex and too time-consuming to be applied in flexible and small-scale production, which often requires the frequent deployment of new applications. Our approach is based on programming by demonstration and enables the programming of finishing operations also for users who are not specialists in robot programming. Programming by demonstration is especially useful for teaching finishing operations because it enables the transfer of expert knowledge about finishing skills to robots without providing lengthy task descriptions or manual coding. Besides the human demonstration of the desired operation, the proposed approach also requires the availability of the kinematic model for the machine tool applied to carry out the finishing operation. We provide several practical examples of grinding and polishing tools and how to integrate them into our approach. Another feature of the proposed system is that user demonstrations of finishing operations can be transferred between different combinations of robots and machine tools.

Assembly tasks performed with a robot often fail due to unforeseen situations, regardless of the fact that we carefully learned and optimized the assembly policy. This problem is even more present in humanoid robots acting in an unstructured environment where it is not possible to anticipate all factors that might lead to the failure of the given task. In this work, we propose a concurrent LfD framework, which associates demonstrated exception strategies to the given context. Whenever a failure occurs, the proposed algorithm generalizes past experience regarding the current context and generates an appropriate policy that solves the assembly issue. For this purpose, we applied PCA on force/torque data, which generates low dimensional descriptor of the current context. The proposed framework was validated in a peg-in-hole (PiH) task using Franka-Emika Panda robot.

Dynamic movement primitives (DMPs) have proven to be an effective movement representation for motor skill learning. In this paper, we propose a new approach for training deep neural networks to synthesize dynamic movement primitives. The distinguishing property of our approach is that it can utilize a novel loss function that measures the physical distance between movement trajectories as opposed to measuring the distance between the parameters of DMPs that have no physical meaning. This was made possible by deriving differential equations that can be applied to compute the gradients of the proposed loss function, thus enabling an effective application of backpropagation to optimize the parameters of the underlying deep neural network. While the developed approach is applicable to any neural network architecture, it was evaluated on two different architectures based on encoder-decoder networks and convolutional neural networks. Our results show that the minimization of the proposed loss function leads to better results than when more conventional loss functions are used.

Learning to recognize and reproduce handwritten characters is already a challenging task both for humans and robots alike, but learning to do the same thing for characters that can be transformed arbitrarily in space, as humans do when writing on a blackboard for instance, significantly ups the ante from a robot vision and control perspective. In previous work we proposed various different forms of encoder-decoder networks that were capable of mapping raw images of digits to dynamic movement primitives (DMPs) such that a robot could learn to translate the digit images into motion trajectories in order to reproduce them in written form. However, even with the addition of convolutional layers in the image encoder, the extent to which these networks are spatially invariant or equivariant is rather limited. In this paper, we propose a new architecture that incorporates both an image-to-motion encoder-decoder and a spatial transformer in a fully differentiable overall network that learns to rectify affine transformed digits in input images into canonical forms, before converting them into DMPs with accompanying motion trajectories that are finally transformed back to match up with the original digit drawings such that a robot can write them in their original forms. We present experiments with various challenging datasets that demonstrate the superiority of the new architecture compared to our previous work and demonstrate its use with a humanoid robot in a real writing task.

In this paper we propose a new approach for efficient programming of grinding and polishing operation. In the proposed system, the initial policy is performed by a skilled operator and recorded with a passive digitizer. The demonstrated policy comprises both position and force data. The optimal robot execution of the task is provided by applying a virtual mechanism approach, which models the polishing/grinding tool as a serial kinematic chain. By joining the robot and the virtual mechanism in an augmented system, additional degrees of freedom are obtained and redundancy resolution can be applied to optimize the demonstrated motion. Another benefit of the proposed approach is that the same policy can be transferred to different combination of robots and grinding/polishing tools without any modification of the captured motion. The proposed approach requires known contact point between the treated object and the polishing/grinding tool. We propose a novel approach for accurate estimation of this point using data obtained from the force-torque sensor. Finally, the demonstrated path is refined to compensate for inaccurate calibration and different dynamics of a robot and the human demonstrator using iterative learning controller. The proposed method was verified in a real industrial environment.

In this paper we propose a new approach for learning perception-action couplings. We show that by collecting a suitable set of raw images and the associated movement trajectories, a deep encoder-decoder network can be trained that takes raw images as input and outputs the corresponding dynamic movement primitives. We propose suitable cost functions for training the network and describe how to calculate their gradients to enable effective training by back-propagation. We tested the proposed approach both on a synthetic dataset and on a widely used MNIST database to generate handwriting movements from raw images of digits. The calculated movements were also applied for digit writing with a real robot.

Encoding desired motions into dynamic movement primitives (DMPs) is a common way for generating compact task representations that are able to handle sensor-based goal adaptations. At the same time, a robot should not only express adaptive motion capabilities at planning level, but use also contact wrench feedback in the adaptation and learning process of the DMP. Despite first approaches exist in this direction, no fully integrated approach has been proposed so far. In this paper, we introduce a new class of admittance-coupled DMPs that addresses environmental changes by including contact wrench feedback dynamics into the DMP formalism. Moreover, a novel iterative learning approach is devised that is based on monitoring the overall system passivity analysis in terms of reference power tracking. Simulations and experimental results with the Kuka LWR robot maintaining a non-rigid contact with the environment (wiping a surface) are shown for supporting the validity of our approach.

Autonomous trajectory generation through generalization requires a database of motion, which can be difficult and time consuming to obtain. In this paper, we propose a method for autonomous expansion of a database for the generation of compliant and accurate motion, achieved through the framework of compliant movement primitives (CMPs). These combine task-specific kinematic and corresponding feed-forward dynamic trajectories. The framework allows for generalization and modulation of dynamic behavior. Inspired by human sensorimotor learning abilities, we propose a novel method that can autonomously learn task-specific torque primitives (TPs) associated to given kinematic trajectories, encoded as dynamic movement primitives. The proposed algorithm is completely autonomous, and can be used to rapidly generate and expand the CMP database. Since CMPs are parameterized, statistical generalization can be used to obtain an initial TP estimate of a new CMP. Thereby, the learning rate of new CMPs can be significantly improved. The evaluation of the proposed approach on a Kuka LWR-4 robot performing a peg-in-hole task shows fast TP acquisition and accurate generalization estimates in real-world scenarios.

We present a three-level cognitive system in a learning by demonstration context. The system allows for learning and transfer on the sensorimotor level as well as the planning level. The fundamentally different data structures associated with these two levels are connected by an efficient mid-level representation based on so-called “semantic event chains.” We describe details of the representations and quantify the effect of the associated learning procedures for each level under different amounts of noise. Moreover, we demonstrate the performance of the overall system by three demonstrations that have been performed at a project review. The described system has a technical readiness level (TRL) of 4, which in an ongoing follow-up project will be raised to TRL 6.

In this study we propose a new method to enhance the performance of iterative learning control (ILC). We focus on robotic tasks dealing with adaptation to the unknown or partially known environment, where the robot has to learn the environment geometry in order to perform the desired task with the given reference forces and torques. The initial motion trajectories are obtained by kinesthetic teaching, whereas the required forces and torques are prescribed by the task. We are interested in incremental learning, which assures smooth and safe operation, aiming at handling of delicate, fragile objects, such as objects made of glass. In order to achieve these goals we propose a new adaptive ILC scheme, where the adaptation is supervised by reinforcement learning. We also show how to apply ILC to orientational motion, taking into account the curved geometry of SO(3). The performance of the proposed algorithm is verified on a bi-manual glass wiping task.

Continuous and on-line adaptation of robotic trajectories is one of the key properties of robotic control policies that make robots useful in unstructured environments, where the conditions of the external world are not predefined or stationary. Modification of robotic trajectories through human intervention, in order to make them more suitable to the user, is often termed as robotic coaching. In the manner of a tutor, the user shows to the robot which part of the motion to change and how. Predefined gestures acquired with visual systems and direct physical interaction are two possibilities to realize robotic coaching. In this paper we study what are the desired user features and which of the three tested coaching methods is deemed most favorable by a group of 7 subjects. The three methods we explore all build on on-line coaching of dynamic motion primitives, but are based on different feedback signals, i. e., visual or force feedback, and different low-level robot control approaches, i. e., a stiff or a compliant robot. The experiment required coaching of a KUKA LWR-4 robot arm while wiping a flat surface, so that it followed two simple patterns. The survey presented in this paper aims at providing the designers of human-robot coaching interfaces with answers on the feasibility, advantages and drawbacks of the three methods for on-line coaching analyzed in this paper: 1. visual feedback, 2. force feedback with a stiff robot, and 3. position feedback with a compliant robot.

An effective robot trajectory representation should encode all relevant aspects of the desired motion. For kinematic representations, this means that both the spatial course of the trajectory and its speed profile must be specified. The concept of dynamic movement primitives (DMP) provides a kinematic representation that fully specifies these two aspects of motion. They are, however, not separated from each other within the DMP representation. This can be problematic when movements with significant speed variations are compared within movement recognition and skill learning algorithms. In such comparisons it is often important to distinguish between the spatial and temporal aspects of motion. In this paper we propose a new representation based on dynamic movement primitives, where spatial and temporal aspects are well separated. We demonstrate the effectiveness of the proposed representation for statistical learning of robot skills and movement recognition and compare the performance with standard DMPs, where temporal and spatial aspects of motion are intertwined.

Human-demonstrated motion transferred to a robotic platform often needs to be adapted to the current state of the environment or to modified task requirements. Adaptation, i. e. learning of a modified behavior, needs to be fast to enable quick utilization of the robot either in industry or in future household-assistant tasks. In this paper we show how to accelerate trajectory adaptation based on learning of coupling terms in the framework of dynamic movement primitives (DMPs). Our method applies ideas from feedback error learning to iterative learning control (ILC). By taking into account the actual physical constraints of the synchronous motion - through synchronization of both positions (or forces) and velocities - it is not only a more faithful representation of actual real-world processes, but it also accelerates the speed of convergence. To show the applicability of the approach in the framework of DMPs, we tested it on a formulation which encodes an initial discrete motion, followed by a periodic behavior, all in a single system. Modifications of the original discrete-periodic formulation now also allow for the use of DMP temporal scaling property. In the paper we also show how the DMP coupling can be implemented in joint space, whereas the measured forces and previous approaches always remained in the task space. We applied our approach to an example dual-arm synchronization task on Sarcos humanoid robot CB-i.

In this paper we exploit Iterative Learning Controllers (ILC) schemes in force adaptation tasks. We propose to encode the control signal with Radial Basis Functions (RBF), which enhances the robustness of the ILC scheme and allows to vary the execution speed of the learned motion. For that a novel control scheme is proposed, which updates the feedforward compensation signals based on current iteration cycle signals in contrast to the standard ILC, which uses signals from the previous iteration cycle. This reduces the computational burden and enhances the adaptation speed. Stability of the proposed control law is analysed and discussed. The proposed approach is evaluated in simulation and on a Kuka Light Weight Robot Arm where the task was to perform force-based surface following with both discrete and periodic movements.

Just as their discrete counterparts, periodic or rhythmic dynamic motion primitives allow easily modulated and robust motion generation, but for periodic tasks. In this paper we present an approach for modulating periodic dynamic movement primitives based on force feedback, allowing for rich motor behavior and skills. We propose and evaluate the combination of feedback and learned feed-forward terms to fully adapt the motions of a robot in order to achieve a desired force interaction with the environment. For the learning we employ the notion of repetitive control, which can effectively minimize the error of behavior towards a given reference. To demonstrate the approach, we show results of simulated and real world experiments on a compliant humanoid robot COMAN. We show the initial results of utilizing the approach to control a pedal-racer, a demanding balance toy best described as a hybrid between a skateboard and a bicycle.

The creation and adaptation of motor behaviors is an important capability for autonomous robots. In this paper we propose an approach for altering existing robot behaviors online, where a human coach interactively changes the robot motion to achieve the desired outcome. Using hand gestures, the human coach can specify the desired modifications to the previously acquired behavior. To preserve a natural posture while performing the task, the movement is encoded in the robot's joint space using periodic dynamic movement primitives. The coaching gestures are mapped to the robot joint space via robot Jacobian and used to create a virtual force field affecting the movement. A recursive least squares technique is used to modify the existing movement with respect to the virtual force field. The proposed approach was evaluated on a simulated three degrees of freedom planar robot and on a real humanoid robot, where human coaching gestures were captured by an RGB-D sensor. Although our focus was on rhythmic movements, the developed approach is also applicable to discrete (point-to-point) movements.

We present an approach for autonomous interactive object segmentation by a humanoid robot. The visual segmentation of unknown objects in a complex scene is an important prerequisite for e.g. object learning or grasping, but extremely difficult to achieve through passive observation only. Our approach uses the manipulative capabilities of humanoid robots to induce motion on the object and thus integrates the robots manipulation and sensing capabilities to segment previously unknown objects. We show that this is possible without any human guidance or pre-programmed knowledge, and that the resulting motion allows for reliable and complete segmentation of new objects in an unknown and cluttered environment. We extend our previous work, which was restricted to textured objects, by devising new methods for the generation of object hypotheses and the estimation of their motion after being pushed by the robot. These methods are mainly based on the analysis of motion of color annotated 3D points obtained from stereo vision, and allow the segmentation of textured as well as non-textured rigid objects. In order to evaluate the quality of the obtained segmentations, they are used to train a simple object recognizer. The approach has been implemented and tested on the humanoid robot ARMAR-III, and the experimental results confirm its applicability on a wide variety of objects even in highly cluttered scenes.

Dynamic movement primitives (DMPs) were proposed as an efficient way for learning and control of complex robot behaviors. They can be used to represent point-to-point and periodic movements and can be applied in Cartesian or in joint space. One problem that arises when DMPs are used to define control policies in Cartesian space is that there exists no minimal, singularity-free representation of orientation. In this paper we show how dynamic movement primitives can be defined for non minimal, singularity free representations of orientation, such as rotation matrices and quaternions. All of the advantages of DMPs, including ease of learning, the ability to include coupling terms, and scale and temporal invariance, can be adopted in our formulation. We have also proposed a new phase stopping mechanism to ensure full movement reproduction in case of perturbations.

In this paper we address the problem of accurate trajectory tracking while ensuring compliant robotic behaviour for periodic tasks. We propose an approach for on-line learning of task-specific dynamics, i.e. task specific movement trajectories and corresponding force/torque profiles. The proposed control framework is a multi-step process, where in the first step a human tutor shows how to perform the desired periodic task. A state estimator based on an adaptive frequency oscillator combined with dynamic movement primitives is employed to extract movement trajectories. In the second step, the movement trajectory is accurately executed in the controlled environment under human supervision. In this step, the robot is accurately tracking the acquired movement trajectory, using high feedback gains to ensure accurate tracking. Thus it can learn the corresponding force/torque profiles, i. e. task-specific dynamics. Finally, in the third step, the movement is executed with the learned feedforward task-specific dynamic model, allowing for low position feedback gains, which implies compliant robot behaviour. Thus, it is safe for interaction with humans or the environment. The proposed approach was evaluated on a Kuka LRW robot performing object manipulation and crank turning.

The framework of dynamic movement primitives (DMPs) contains many favorable properties for the execution of robotic trajectories, such as indirect dependence on time, response to perturbations, and the ability to easily modulate the given trajectories, but the framework in its original form remains constrained to the kinematic aspect of the movement. In this paper, we bridge the gap to dynamic behavior by extending the framework with force/torque feedback. We propose and evaluate a modulation approach that allows interaction with objects and the environment. Through the proposed coupling of originally independent robotic trajectories, the approach also enables the execution of bimanual and tightly coupled cooperative tasks. We apply an iterative learning control algorithm to learn a coupling term, which is applied to the original trajectory in a feed-forward fashion and, thus, modifies the trajectory in accordance to the desired positions or external forces. A stability analysis and results of simulated and real-world experiments using two KUKA LWR arms for bimanual tasks and interaction with the environment are presented. By expanding on the framework of DMPs, we keep all the favorable properties, which is demonstrated with temporal modulation and in a two-agent obstacle avoidance task.

Direct transfer of human motion trajectories to humanoid robots does not result in dynamically stable robot movements due to the differences in human and humanoid robot kinematics and dynamics. We developed a system that converts human movements captured by a low-cost RGB-D camera into dynamically stable humanoid movements. The transfer of human movements occurs in real-time. As need arises, the developed system can smoothly transition between unconstrained movement imitation and imitation with balance control, where movement reproduction occurs in the null space of the balance controller. The developed balance controller is based on an approximate model of the robot dynamics, which is sufficient to stabilize the robot during on-line imitation. However, the resulting movements cannot be guaranteed to be optimal because the model of the robot dynamics is not exact. The initially acquired movement is therefore subsequently improved by model-free reinforcement learning, both with respect to the accuracy of reproduction and balance control. We present experimental results in simulation and on a real humanoid robot.

The goal of this study is to provide an architecture for a generic definition of robot manipulation actions. We emphasize that the representation of actions presented here is “procedural”. Thus, we will define the structural elements of our action representations as execution protocols. To achieve this, manipulations are defined using three levels. The toplevel defines objects, their relations and the actions in an abstract and symbolic way. A mid-level sequencer, with which the action primitives are chained, is used to structure the actual action execution, which is performed via the bottom level. This (lowest) level collects data from sensors and communicates with the control system of the robot. This method enables robot manipulators to execute the same action in different situations i.e. on different objects with different positions and orientations. In addition, two methods of detecting action failure are provided which are necessary to handle faults in system. To demonstrate the effectiveness of the proposed framework, several different actions are performed on our robotic setup and results are shown. This way we are creating a library of human-like robot actions, which can be used by higher-level task planners to execute more complex tasks.

The framework of dynamic movement primitives allows the generation of discrete and periodic trajectories, which can be modulated in various aspects. We propose and evaluate a novel modulation approach that includes force feedback and thus allows physical interaction with objects and the environment. The proposed approach also enables the coupling of independently executed robotic trajectories, simplifying the execution of bimanual and tightly coupled cooperative tasks. We apply an iterative learning control algorithm to learn a coupling term, which is applied to the original trajectory in a feed-forward fashion. The coupling term modifies the trajectory in accordance to either the desired position or external force. The strengths of the approach are shown in bimanual or two-agent obstacle avoidance tasks, where no higher level cognitive reasoning or planning are required. Results of simulated and real-world experiments on the ARMAR-III humanoid robot in interaction and object lifting tasks, and on two KUKA LWR robots in a bimanual setting are presented.

When it comes to learning how to manipulate objects from experience with minimal prior knowledge, robots encounter significant challenges. When the objects are unknown to the robot, the lack of prior object models demands a robust feature descriptor such that the robot can reliably compare objects and the effects of their manipulation. In this paper, using an experimental platform that gathers 3-D data from the Kinect RGB-D sensor, as well as push action trajectories from a tracking system, we address these issues using an action-grounded 3-D feature descriptor. Rather than using pose-invariant visual features, as is often the case with object recognition, we ground the features of objects with respect to their manipulation, that is, by using shape features that describe the surface of an object relative to the push contact point and direction. Using this setup, object push affordance learning trials are performed by a human and both pre-push and post-push object features are gathered, as well as push action trajectories. A self-supervised multi-view online learning algorithm is employed to bootstrap both the discovery of affordance classes in the post-push view, as well as a discriminative model for predicting them in the pre-push view. Experimental results demonstrate the effectiveness of self-supervised class discovery, class prediction and feature relevance determination on a collection of unknown objects.

Learning about new objects that a robot sees for the first time is a difficult problem because it is not clear how to define the concept of object in general terms. In this paper we consider as objects those physical entities that are comprised of features which move consistently when the robot acts upon them. Among the possible actions that a robot could apply to a hypothetical object, pushing seems to be the most suitable one due to its relative simplicity and general applicability. We propose a methodology to generate and apply pushing actions to hypothetical objects. A probing push causes visual features to move, which enables the robot to either confirm or reject the initial hypothesis about existence of the object. Furthermore, the robot can discriminate the object from the background and accumulate visual features that are useful for training of state of the art statistical classifiers such as bag of features.

This paper presents a novel approach to control a humanoid head for object tracking. The proposed approach is based on the concept of virtual mechanism, where the real head is enhanced with a virtual link that connects the eye with a point in 3-D space. We tested our implementation on a humanoid head with seven degrees of freedom and two rigidly connected cameras in each eye (wide-angle and telescopic). The experimental results show that the proposed control algorithm can be used to maintain the view of an observed object in the foveal (telescopic) image using information from the peripheral view. Unlike other methods proposed in the literature, our approach shows how to exploit the redundancy of the robot head. The proposed technique is systematic and can be easily implemented on different types of active humanoid heads. The results show good tracking performance regardless of the distance between the object and the head. Moreover, the uncertainties in the kinematic model of the head do not affect the performance of the system.

Acquisition of new sensorimotor knowledge by imitation is a promising paradigm for robot learning. To be effective, action learning should not be limited to direct replication of movements obtained during training but must also enable the generation of actions in situations a robot has never encountered before. This paper describes a methodology that enables the generalization of the available sensorimotor knowledge. New actions are synthesized by the application of statistical methods, where the goal and other characteristics of an action are utilized as queries to create a suitable control policy, taking into account the current state of the world. Nonlinear dynamic systems are employed as a motor representation. The proposed approach enables the generation of a wide range of policies without requiring an expert to modify the underlying representations to account for different task-specific features and perceptual feedback. The paper also demonstrates that the proposed methodology can be integrated with an active vision system of a humanoid robot. 3-D vision data are used to provide query points for statistical generalization. While 3-D vision on humanoid robots with complex oculomotor systems is often difficult due to the modeling uncertainties, we show that these uncertainties can be accounted for by the proposed approach.

In this paper we study object recognition on a humanoid robotic head. The head is equipped with a stereo vision system with two cameras in each eye, where the cameras have lenses with different view angles. Such a system models the foveated structure of a human eye. To facilitate the pursuit of moving objects, we provide mathematical analysis that enables the robot to guide the narrow-view cameras toward the object of interest based on information extracted from the wider views. Images acquired by narrow-view cameras, which produce object images at higher resolutions, are used for recognition. The proposed recognition approach is view-based and is built around a classifier using nonlinear multi-class support vector machines with a special kernel function. We show experimentally that the increased resolution leads to higher recognition rates.

In this video presentation we introduce a 50 degrees of freedom humanoid robot, CB -ComputationalBrain[1]. CB is a humanoid robot created for exploring the underlying processing of the human brain while dealing with the real world. We place our investigations within real world contexts, as humans do. In so doing, we focus on utilising a system that is closer to humans - in sensing, kinematics configuration and performance. We present a full-body compliance controller that was developed for the motion control of our humanoid robot [2]. Our initial experimentation on our system includes: 1) full-body compliant control - physical interactions/balancing/motion control; 2) the integrated visual ocular-motor responses; 3) perception and control - reaching, foveation, and active object recognition; 4) our studies of Central Pattern Generator for walking.

In this paper, we present an image-based markerless human motion capture system, intended for humanoid robot systems. The restrictions set by this ambitious goal are numerous. The input of the system is a sequence of stereo image pairs only, captured by cameras positioned at approximately eye distance. No artificial markers can be used to simplify the estimation problem. Furthermore, the complexity of all algorithms incorporated must be suitable for real-time application, which is maybe the biggest problem when considering the high dimensionality of the search space. Finally, the system must not depend on a static camera setup and has to find the initial configuration automatically. We present a system, which tackles these problems by combining multiple cues within a particle filter framework, allowing the system to recover from wrong estimations in a natural way. We make extensive use of the benefit of having a calibrated stereo setup. To reduce search space implicitly, we use the 3D positions of the hands and the head, computed by a separate hand and head tracker using a linear motion model for each entity to be tracked. With stereo input image sequences at a resolution of 320 times 240 pixels, the processing rate of our system is 15 Hz on a 3 GHz CPU. Experimental results documenting the performance of our system are available in form of several videos.

In this paper we discuss active humanoid vision systems that realize foveation using two rigidly connected cameras in each eye. We present an exhaustive analysis of the relationship between the positions of the observed point in the foveal and peripheral view with respect to the intrinsic and extrinsic parameters of both cameras and 3-D point position. Based on these results we propose a control scheme that can be used to maintain the view of the observed object in the foveal image using information from the peripheral view. Experimental results showing the effectiveness of the proposed foveation control are also provided

Object recognition requires a robot to perform a number of nontrivial tasks such as finding objects of interest, directing its eyes towards the objects, pursuing them, and identifying the objects once they appear in the robot's central vision. We have recently developed a recognition system on a humanoid robot, which makes use of foveated vision to accomplish these tasks (A Ude, et al., 2003). In this paper we present several substantial improvements to this system. We present a biologically motivated object representation scheme based on Gabor kernel functions and show how to employ support vector machines to identify known objects in foveal images based on this representation. A mechanism for visual search is integrated into the system to find objects of interest in peripheral images. The framework also includes a control scheme for eye movements, which are directed using the results of attentive processing in peripheral images.

We seek intuitive, efficient ways to create and direct human-like behaviors for humanoid robots. Here we present a method to enable humanoid robots to acquire movements by imitation. The robot uses 3D vision to perceive the movements of a human teacher, and then estimates the teacher's body postures using a fast full-body inverse kinematics method that incorporates a kinematic model of the teacher. This solution is then mapped to the robot and reproduced in real-time. The robustness of the method is tested on a 30-degree-of-freedom Sarcos humanoid robot located at ATR using 3D vision data from external cameras and from head-mounted cameras.

In this paper we present a humanoid system that can integrate information provided by its foveal and peripheral cameras. We use peripheral vision to detect and pursue objects of interest based on simple shape and color models. A detection event triggers the robot to direct its eyes towards the object, thus making a more detailed analysis of the observed objects in higher resolution foveal images feasible. The recognition is based on principal component analysis and is performed while the robot actively pursues the detected object. The classification results are inferred using information from a video stream rather than just a single image. Once the desired object is recognized, the robot reaches for it while ignoring other objects.

This paper describes humanoid robot learning from observation and game playing using information provided by a real-time PC-based vision system. To cope with extremely fast motions that arise in the environment, a visual system capable of perceiving the motion of several objects at 60 fields per second was developed. We have designed a suitable error recovery scheme for our vision system to ensure successful game playing over longer periods of time. To increase the learning rate of the robot it is given domain knowledge in the form of primitives. The robot learns how to perform primitives from data collected while observing a human. The robot control system and primitive use strategy are also explained.

We describe a real-time visual system that enables a humanoid robot to learn from and interact with humans. The core of the visual system is a probabilistic tracker that uses shape and color information to find relevant objects in the scene. Multiscale representations, windowing and masking are employed to accelerate the data processing. The perception system is directly coupled with the motor control system of our humanoid robot DB. We present an example of on-line interaction with a humanoid robot: mimicking of human hand motion. The generation of humanoid robot motion based on the human motion is accomplished in real-time. The study is supported by experimental results on DB.

The formulation and optimization of joint trajectories for humanoid robots is quite different from this same task for standard robots because of the complexity of the humanoid robots' kinematics. We exploit the similarity between the movements of a humanoid robot and human movements to generate joint trajectories for such robots. In particular we show how to transform human motion information captured by an optical tracking device into a high dimensional trajectory of a humanoid robot. We utilize B-spline wavelets to efficiently represent the joint trajectories and to automatically select the density of the basis functions on the time axis. We applied our method to the task of teaching a humanoid robot how to make a dance movement.

Addresses the problem of estimating human body motion from video. Its main contribution is the introduction of a robust optimization framework that leads to reliable and accurate body tracking and posture recovery. The proposed approach is resistant to occlusions and demonstrates that it is possible to treat different problems arising in human motion analysis in a unified way without using many decision thresholds. The implemented system requires only a standard CCD camera and no special markers on the body. We present experimental results showing the reliability of the implemented tracker.

Pose estimation from an arbitrary number of 2D to 3D feature correspondences is often done by minimising a nonlinear criterion function using one of the minimal representations for the orientation. However, there are many advantages in using unit quaternions to represent the orientation. However, a straight forward formulation of the pose estimation problem based on quaternions results in a constrained optimisation problem. In this paper we propose a new method for solving general nonlinear least squares optimisation problems involving unit quaternion functions based on unconstrained optimisation techniques. We demonstrate the effectiveness of our approach for pose estimation from 2D to 3D line segment correspondences.

A strategy for the fusion of information from a stereo image pair for model-based object recognition is discussed. Our scheme combines a new method for feature grouping with a region-based stereo matching and a hypothesize-and-verify paradigm. The grouping method developed is based on a graph theoretical algorithm. It exploits prior knowledge to find the groups of image features which are likely to come from a sought model(s). The Bayesian classification is used to deal with the resulting hypotheses. A mechanism for a dynamic threshold modification is incorporated into the system to enable the grouping at different resolutions. Unlike classical techniques for object recognition from stereo, our strategy does not depend on a data driven computation of a depth map. We argue that a propulsive reconstruction of 3D information can be more efficient and robust.