In recent years, depth sensors have become more and more affordable and have found their way into a growing amount of robotic systems. However, mono- or multi-modal sensor registration, often a necessary step for further pro-cessing, faces many challenges on raw depth images or point clouds. This paper presents a method of converting depth data into images capable of visualizing spatial details that are basically hidden in traditional depth images. After noise removal, a neighborhood of points forms two normal vectors whose difference is encoded into this new conversion. Compared to Bearing Angle images, our method yields brighter, higher-contrast images with more visible contours and more details. We tested feature-based pose estimation of both conversions in a visual odometry task and RGB-D SLAM. For all tested features, AKAZE, ORB, SIFT, and SURF, our new Flexion images yield better results than Bearing Angle images and show great potential to bridge the gap between depth data and classical computer vision. Source code is available here: https://rlsch.github.io/depth-flexion-conversion.

Many robotic tasks require fast and accurate force sensing capabilities to ensure adaptive behavior execution. While dedicated force-torque (FT) sensors are a common option, such devices induce extra costs, need additional power supply, and add weight to otherwise light-weight robotic systems. This paper presents a machine learning approach for estimating external forces acting on a robot based on common internal sensors only. In the training phase, a behavior-specific proprioceptive model is learned as compact representation of the expected proprioceptive feedback during task execution. First, the proprioceptive sensors relevant for the given behavior are identified using information-theoretic measures. Then, the proprioceptive model is learned using deep learning techniques. During behavior execution, the proprioceptive model is applied to actual sensor readings for estimation of external forces. Experiments performed with the UR5 robot demonstrate the ability for fast and accurate force estimation even in situations where a dedicated commercial FT sensor is not applicable.

Objective assessment in long-term rehabilitation under real-life recording conditions is a challenging task. We propose a data-driven method to evaluate changes in motor function under uncontrolled, long-term conditions with the low-cost Microsoft Kinect sensor. Instead of using human ratings as ground truth data, we propose kinematic features of hand motion, healthy reference trajectories derived by principal component regression, and methods taken from machine learning to analyze the progression of motor function. We demonstrate the capability of this approach on datasets with repetitive unrestrained bi-manual drumming movements in three-dimensional space of stroke survivors, patients suffering of Parkinson's disease, and a healthy control group. We present processing steps to eliminate the influence of varying recording setups under real-life conditions and offer visualization methods to support clinicians in the evaluation of treatment effects.

Underground mines are a dangerous working environment and, therefore, robots could help putting less humans at risk. Traditional robots, sensors, and software often do not work reliably underground due to the harsh environment. This paper analyzes requirements and presents a robot design capable of navigating autonomously underground and manipulating objects with a robotic arm. The robot's base is a robust four wheeled platform powered by electric motors and able to withstand the harsh environment. It is equipped with color and depth cameras, lighting, laser scanners, an inertial measurement unit, and a robotic arm. We conducted two experiments testing mapping and autonomous navigation. Mapping a 75 meters long route including a loop closure results in a map that qualitatively matches the original map to a good extent. Testing autonomous driving on a previously created map of a second, straight, 150 meters long route was also successful. However, without loop closure, rotation errors cause apparent deviations in the created map. These first experiments showed the robot's operability underground.

We present a data-driven imitation learning system for learning human-robot interactions from human-human demonstrations. During training, the movements of two interaction partners are recorded through motion capture and an interaction model is learned. At runtime, the interaction model is used to continuously adapt the robot's motion, both spatially and temporally, to the movements of the human interaction partner. We show the effectiveness of the approach on complex, sequential tasks by presenting two applications involving collaborative human-robot assembly. Experiments with varied object hand-over positions and task execution speeds confirm the capabilities for spatio-temporal adaption of the demonstrated behavior to the current situation.

Robotic manipulation tasks often require the control of forces and torques exerted on external objects. This paper presents a machine learning approach for estimating forces when no force sensors are present on the robot platform. In the training phase, the robot executes the desired manipulation tasks under controlled conditions with systematically varied parameter sets. All internal sensor data, in the presented case from more than 100 sensors, as well as the force exerted by the robot are recorded. Using Transfer Entropy, a statistical model is learned that identifies the subset of sensors relevant for torque estimation in the given task. At runtime, the model is used to accurately estimate the torques exerted during manipulations of the demonstrated kind. The feasibility of the approach is shown in a setting where a robotic manipulator operates a torque wrench to fasten a screw nut. Torque estimates with an accuracy of well below ±1Nm are achieved. A strength of the presented model is that no prior knowledge of the robot's kinematics, mass distribution or sensor instrumentation is required.

In order to ensure safe operation, robots must be able to reliably detect behavior perturbations that result from unexpected physical interactions with their environment and human co-workers. While some robots provide firmware force sensors that generate rough force estimates, more accurate force measurements are usually achieved with dedicated force-torque sensors. However, such sensors are often heavy, expensive and require an additional power supply. In the case of lightweight manipulators, the already limited payload capabilities may be reduced in a significant way. This paper presents an experience-based approach for accurately estimating external forces being applied to a robot without the need for a force-torque sensor. Using Information Transfer, a subset of sensors relevant to the executed behavior are identified from a larger set of internal sensors. Models mapping robot sensor data to force-torque measurements are learned using a neural network. These models can be used to predict the magnitude and direction of perturbations from affordable, proprioceptive sensors only. Experiments with a UR5 robot show that our method yields force estimates with accuracy comparable to a dedicated force-torque sensor. Moreover, our method yields a substantial improvement in accuracy over force-torque values provided by the robot firmware.

Abstract Interaction meshes are a promising approach for generating natural behaviors of virtual characters during ongoing user interactions. In this paper, we propose several extensions to the interaction mesh approach based on statistical analyses of the underlying example interactions. By applying principal component analysis and correlation analysis in addition to joint distance calculations, both the interaction mesh topology and the constraints used for mesh optimization can be generated in an automated fashion that accounts for the spatial and temporal contexts of the interaction. Copyright © 2015 John Wiley & Sons, Ltd.

In many settings, e.g. physical human-robot interaction, robotic behavior must be made robust against more or less spontaneous application of external forces. Typically, this problem is tackled by means of special purpose force sensors which are, however, not available on many robotic platforms. In contrast, we propose a machine learning approach suitable for more common, although often noisy sensors. This machine learning approach makes use of Dynamic Mode Decomposition (DMD) which is able to extract the dynamics of a nonlinear system. It is therefore well suited to separate noise from regular oscillations in sensor readings during cyclic robot movements under different behavior configurations. We demonstrate the feasibility of our approach with an example where physical forces are exerted on a humanoid robot during walking. In a training phase, a snapshot based DMD model for behavior specific parameter configurations is learned. During task execution the robot must detect and estimate the external forces exerted by a human interaction partner. We compare the DMD-based approach to other interpolation schemes and show that the former outperforms the latter particularly in the presence of sensor noise. We conclude that DMD which has so far been mostly used in other fields of science, particularly fluid mechanics, is also a highly promising method for robotics.

In this paper we present a new approach for learning responsive robot behavior by imitation of human interaction partners. Extending previous work on robot imitation learning, that has so far mostly concentrated on learning from demonstrations by a single actor, we simultaneously record the movements of two humans engaged in on-going interaction tasks and learn compact models of the interaction. Extracted interaction models can thereafter be used by a robot to engage in a similar interaction with a human partner. We present two algorithms for deriving interaction models from motion capture data as well as experimental results on a humanoid robot.

In order for humans and robots to engage in direct physical interaction several requirements have to be met. Among others, robots need to be able to adapt their behavior in order to facilitate the interaction with a human partner. This can be achieved using machine learning techniques. However, most machine learning scenarios to-date do not address the question of how learning can be achieved for tightly coupled, physical touch interactions between the learning agent and a human partner. This paper presents an example for such human in-the-loop learning scenarios and proposes a computationally cheap learning algorithm for this purpose. The efficiency of this method is evaluated in an experiment, where human care givers help an android robot to stand up.

Realistic behavior of control actuators is important for virtual proto-typing applications. We present a systematic approach for modeling such articulated components as described in the European Standard EN 894-3. Control actuators may have several rotational and translational degrees of freedom (DOFs), possibly with discrete lock states. During user interactions, information about the actuators' manipulation is collected and made available to the higher application layers in the form of interaction events. This allows for recording and playback of demonstrated manipulation sequences for many purposes, such as ergonomics evaluations involving virtual humans. The framework uses XML for declaration and is implemented using a freely available physics engine.

Abstract We propose a novel data‐driven animation method for the synthesis of natural looking human grasping. Motion data captured from human grasp actions is used to train a probabilistic model of the human grasp space. This model greatly reduces the high number of degrees of freedom of the human hand to a few dimensions in a continuous grasp space. The low dimensionality of the grasp space in turn allows for efficient optimization when synthesizing grasps for arbitrary objects. The method requires only a short training phase with no need for preprocessing of graphical objects for which grasps are to be synthesized. Copyright © 2008 John Wiley & Sons, Ltd.

This paper presents a comparison of various classification methods for the problem of recognizing grasp types involved in object manipulations performed with a data glove. Conventional wisdom holds that data gloves need calibration in order to obtain accurate results. However, calibration is a time-consuming process, inherently user-specific, and its results are often not perfect. In contrast, the present study aims at evaluating recognition methods that do not require prior calibration of the data glove, by using raw sensor readings as input features and mapping them directly to different categories of hand shapes. An experiment was carried out, where test persons wearing a data glove had to grasp physical objects of different shapes corresponding to the various grasp types of the Schlesinger taxonomy. The collected data was analyzed with 28 classifiers including different types of neural networks, decision trees, Bayes nets, and lazy learners. Each classifier was analyzed in six different settings, representing various application scenarios with differing generalization demands. The results of this work are twofold: (1) We show that a reasonably well to highly reliable recognition of grasp types can be achieved - depending on whether or not the glove user is among those training the classifier - even with uncalibrated data gloves. (2) We identify the best performing classification methods for recognition of various grasp types. To conclude, cumbersome calibration processes before productive usage of data gloves can be spared in many situations.

We present a novel method for virtual character animation that we call action capture. In this approach, virtual characters learn to imitate the actions of Virtual Reality (VR) users by tracking not only the users' movements but also their interactions with scene objects.Action capture builds on conventional motion capture but differs from it in that higher-level action representations are transferred rather than low-level motion data. As an advantage, the learned actions can often be naturally applied to varying situations, thus avoiding retargetting problems of motion capture. The idea of action capture is inspired by human imitation learning; related methods have been investigated for a longer time in robotics. The paper reviews the relevant literature in these areas before framing the concept of action capture in the context of VR-based character animation. We also present an example in which the actions of a VR user are transferred to a virtual worked.

Recently videoconferencing has been extended from human face-to-face communication to human machine interaction with Virtual Environments [2]. Relying on established videoconferencing protocol standards this thin client solution does not require specialized 3D soft- or hardware and scales well to multimedia enabled mobile devices. This brings a whole range of new applications to the mobile platform. To facilitate our research in mobile interaction the Open Source project P@CE has been started to bring a full-featured videoconferencing client to the Pocket PC platform.

Detailed animation of 3D articulated body models is in principle desirable but is also a highly resource-intensive task. Resource limitations are particularly critical in 3D visualizations of multiple characters in real-time game sequences. We investigated to what extent observers perceptually process the level of detail in naturalistic character animations. Only if such processing occurs would it be justified to spend valuable resources on richness of detail. An experiment was designed to test the effectiveness of 3D body animation. Observers had to judge the level of overall skill exhibited by four simulated soccer teams. The simulations were based on recorded RoboCup simulation league games. Thus objective skill levels were known from the teams' placement in the tournament. The animations' level of detail was varied in four increasing steps of modelling complexity. Results showed that observers failed to notice the differences in detail. Nonetheless, clear effects of character animation on perceived skill were found. We conclude that character animation co-determines perceptual judgements even when observers are completely unaware of these manipulations. Copyright © 2000 John Wiley & Sons, Ltd.