We introduce a novel gradient-based approach for solving sequential tasks by dynamically adjusting the underlying myopic potential field in response to feedback and the world's regularities. This adjustment implicitly considers subgoals encoded in these regularities, enabling the solution of long sequential tasks, as demonstrated by solving the traditional planning domain of Blocks World–without any planning. Unlike conventional planning methods, our feedbackdriven approach adapts to uncertain and dynamic environments, as demonstrated by one hundred real-world trials involving drawer manipulation. These experiments highlight the robustness of our method compared to planning and show how interactive perception and error recovery naturally emerge from gradient descent without explicitly implementing them. This offers a computationally efficient alternative to planning for a variety of sequential tasks, while aligning with observations on biological problem-solving strategies.

Visual uncertainties such as occlusions, lack of texture, and noise present significant challenges in obtaining accurate kinematic models for safe robotic manipulation. We introduce a probabilistic real-time approach that leverages the human hand as a prior to mitigate these uncertainties. By tracking the constrained motion of the human hand during manipulation and explicitly modeling uncertainties in visual observations, our method reliably estimates an object's kinematic model online. We validate our approach on a novel dataset featuring challenging objects that are occluded during manipulation and offer limited articulations for perception. The results demonstrate that by incorporating an appropriate prior and explicitly accounting for uncertainties, our method produces accurate estimates, outperforming two recent baselines by 195 % and 140 %, respectively. Furthermore, we demonstrate that our approach's estimates are precise enough to allow a robot to manipulate even small objects safely.

A robotic system’s hardware and control policy must be co-optimized to ensure they complement each other to interact robustly with the environment. However, this combined search is extremely high-dimensional and intractable without a suitable underlying representation. This paper uses environmental constraints to structure the co-design space for manipulation. We show that task-relevant constraints encode regions of the search space containing reasonable co-design solutions. Furthermore, this underlying representation renders a co-design space amenable to gradient-based optimization. For efficient search, we present the co-design Jacobian that describes how the robot’s motion varies with control as well as hardware design changes. This Jacobian exploits the structure induced by environmental constraints for iterative design updates in the co-design space. Using these two conceptual tools, we co-design manipulators, grippers, and multi-fingered hands, showing that environmental constraints are an effective representation for co-designing diverse manipulation systems. Our methodology also scales well with increased co-design parameters, rendering the co-design of complex, high-dimensional manipulation systems feasible.

Robots need to understand articulated objects, such as drawers. The state of articulated structures is commonly estimated using vision, but visual perception is limited when objects are occluded, have few salient features, or are not in the camera's field of view. Audio sensing does not face these challenges, since sound propagates in a fundamentally different way than light. Therefore we propose to fuse vision and audio sensing to overcome the challenges faced by vision alone. We estimate motion in several drawers and show that an audio-visual approach estimates drawer motion more reliably than only vision – even in settings where the purely visual approach completely breaks down. Additionally, we perform an in-depth analysis of the regularities that govern how motion in drawers shapes their sound.

We present a robust method to visually segment scenes into objects based on motion and appearance. Both these cues provide complementary information that we fuse using two interconnected recursive estimators: One estimates object segmentation from motion as a probabilistic clustering of tracked 3D points, and the other estimates object segmentation from appearance as a probabilistic image segmentation. The interconnected estimators provide a probabilistic and consistent object segmentation in real time, which makes them well suited for many downstream robotic tasks. We evaluate our method on one such task, kinematic structure estimation, on a dataset of interactions with articulated objects and show that our fusion improves object segmentation by 70% and in turn estimated kinematic joints by 26% over a purely motion-based approach. Furthermore, we show the necessity of probabilistic modeling for downstream robotic tasks, achieving 339% of the performance of a recent multimodal but deterministic RNN for object segmentation on the estimation of kinematic structure.

We introduce a Learning from Demonstration (LID) approach for contact-rich manipulation tasks, i.e., tasks in which the manipulandum's motion is constrained by contact with the environment. Our approach is motivated by the insight that even a large number of demonstrations will often not contain sufficient information to obtain a general policy for the task. To obtain general policies, our approach augments the information contained in a single demonstration. This autonomous augmentation is based on the insight that environmental constraints play a central role in generalization. We validate our approach in real-world experiments with mechanisms with multiple, interdependent articulations, including latch locks, chain locks, and drawers with handles. The extracted policies, obtained from a single augmented human demonstration, generalize to different mechanisms of the same type and in varying environmental settings.

We present a simple approach to in-hand cube reconfiguration. By simplifying planning, control, and perception as much as possible, while maintaining robust and general performance, we gain insights into the inherent complexity of in-hand cube reconfiguration. We also demonstrate the effectiveness of combining GOFAI-based planning with the exploitation of environmental constraints and inherently compliant end-effectors in the context of dexterous manipulation. The proposed system outperforms a substantially more complex system for cube reconfiguration based on deep learning and accurate physical simulation, contributing arguments to the discussion about what the most promising approach to general manipulation might be. Project website: https://rbo.gitlab-pages.tu-berlin.de/robotics/simpleIHM/

This paper presents a feedback-control framework for in-hand manipulation (IHM) with dexterous soft hands that enables the acquisition of manipulation skills in the real-world within minutes. We choose the deformation state of the soft hand as the control variable. To control for a desired deformation state, we use coarsley approximated Jacobians of the actuation-deformation dynamics. These Jacobian are obtained via explorative actions. This is enabled by the self-stabilizing properties of compliant hands, which allow us to use linear feedback control in the presence of complex contact dynamics. To evaluate the effectiveness of our approach, we show the generalization capabilities for a learned manipulation skill to variations in object size by 100 %, 360 degree changes in palm inclination and to disabling up to 50 % of the involved actuators. In addition, complex manipulations can be obtained by sequencing such feedback-skills.

Robotic manipulation behavior should be robust to disturbances that violate high-level task-structure. Such robustness can be achieved by constantly monitoring the environment to observe the discrete high-level state of the task. This is possible because different phases of a task are characterized by different sensor patterns and by monitoring these patterns a robot can decide which controllers to execute in the moment. This relaxes assumptions about the temporal sequence of those controllers and makes behavior robust to unforeseen disturbances. We implement this idea as probabilistic filter over discrete states where each state is direcly associated with a controller. Based on this framework we present a robotic system that is able to open a drawer and grasp tennis balls from it in a surprisingly robust way.

Soft robotics is an emerging field that yields promising results for tasks that require safe and robust interactions with the environment or with humans, such as grasping, manipulation, and human-robot interaction. Soft robots rely on intrinsically compliant components and are difficult to equip with traditional, rigid sensors which would interfere with their compliance. We propose a highly flexible tactile sensor that is low-cost and easy to manufacture while measuring contact pressures independently from 14 taxels. The sensor is built from piezoresistive fabric for highly sensitive, continuous responses and from a custom-designed flexible printed circuit board which provides a high taxel density. From these taxels, location and intensity of contact with the sensor can be inferred. In this paper, we explain the design and manufacturing of the proposed sensor, characterize its input-output relation, evaluate its effects on compliance when equipped to the silicone-based pneumatic actuators of the soft robotic RBO Hand 2, and demonstrate that the sensor provides rich and useful feedback for learning-based in-hand object recognition.

A gaze-fixating robot perceives distance to the fixated object and relative positions of surrounding objects immediately, accurately, and robustly. We show how fixation, which is the act of looking at one object while moving, exploits regularities in the geometry of 3D space to obtain this information. These regularities introduce rotation-translation couplings that are not commonly used in structure from motion. To validate, we use a Franka Emika Robot with an RGB camera. We a) find that error in distance estimate is less than 5 mm at a distance of 15 cm, and b) show how relative position can be used to find obstacles under challenging scenarios. We combine accurate distance estimates and obstacle information into a reactive robot behavior that is able to pick up objects of unknown size, while impeded by unforeseen obstacles.

We create a virtual 2D tactile array for soft pneumatic actuators using embedded audio components. We detect contact-specific changes in sound modulation to infer tactile information. We evaluate different sound representations and learning methods to detect even small contact variations. We demonstrate the acoustic tactile sensor array by the example of a PneuFlex actuator and use a Braille display to individually control the contact of 29 x 4 pins with the actuator's 90 x 10 mm palmar surface. Evaluating the spatial resolution, the acoustic sensor localizes edges in x- and y-direction with a root-mean-square regression error of 1.67 mm and 0.0 mm, respectively. Even light contacts of a single Braille pin with a lifting force of 0.17 N are measured with high accuracy. Finally, we demonstrate the sensor's sensitivity to complex contact shapes by successfully reading the 26 letters of the Braille alphabet from a single display cell with a classification rate of 88 %.

In this article, we present theRBO Hand 3, a highly capable and versatile anthropomorphic soft hand based on pneumatic actuation. TheRBO Hand 3is designed to enable dexterous manipulation, to facilitate transfer of insights about human dexterity, and to serve as a robust research platform for extensive real-world experiments. It achieves these design goals by combining many degrees of actuation with intrinsic compliance, replicating relevant functioning of the human hand, and by combining robust components in a modular design. TheRBO Hand 3possesses 16 independent degrees of actuation, implemented in a dexterous opposable thumb, two-chambered fingers, an actuated palm, and the ability to spread the fingers. In this article, we derive the design objectives that are based on experimentation with the hand’s predecessors, observations about human grasping, and insights about principles of dexterity. We explain in detail how the design features of theRBO Hand 3achieve these goals and evaluate the hand by demonstrating its ability to achieve the highest possible score in the Kapandji test for thumb opposition, to realize all 33 grasp types of the comprehensive GRASP taxonomy, to replicate common human grasping strategies, and to perform dexterous in-hand manipulation.

This paper offers an explanation of why humans can effortlessly grasp objects from a pile. We identified a regularity in objects’ motion when pushed, namely, an object separates and stabilizes in front of the pusher. We devise an open-loop grasping strategy leveraging this regularity in piles of nearly identical objects. Our real robot robustly grasps round objects beside a wall with success rates between 95% and 100% without visual or tactile feedback. We analyze our grasping strategy extensively both in real-world and simulated experiments. We observe that object roundness improves grasping and the motion pattern also manifests in small piles beside a wall. Our qualitative simulation can approximate the real robot’s grasping behavior, and we apply open-loop grasping in an warehouse pick-and-place application.

We present an active acoustic sensor that turns soft pneumatic actuators into contact sensors. The whole surface of the actuator becomes a sensor, rendering the question of where best to place a contact sensor unnecessary. At the same time, the compliance of the soft actuator remains unaffected. A small, embedded speaker emits a frequency sweep which travels through the actuator before it is recorded with an embedded microphone. The specific contact state of the actuator affects how the sound is modulated while traversing the structure. We learn to recognize these changes in the sound and map them to the corresponding contact locations. We demonstrate the method on the PneuFlex actuator. The active acoustic sensor achieves a classification rate of 93% and mean regression error of 3.7mm. It is robust against background noises and different objects. Finally, we test it on a Panda robot arm and show that it is unaffected by motor noises and other active sensors.

The space of all possible deformations of soft robotic actuators is extremely large. It is impossible to explicitly measure each internal degree of freedom, regardless of the number and types of sensors. It is, however, possible to measure a smaller subset of task-relevant deformations using only a few well-placed sensors. But for a different task, the soft actuator's deformation behavior might differ significantly. Instead of finding a new sensor placement for the new task, which would result in a separate hand for every task, we propose a method that maintains the original sensors and uses prior knowledge about each task to extend the applicability of the existing sensorized actuators to new tasks. We demonstrate our approach by the example of a PneuFlex actuator of the RBO Hand 2. When sensorizing the actuator for a single task, the sensor model does not transfer well to other tasks. Using our multi-task method, we train new sensor models that use prior knowledge about the tasks. The new models improve measurement accuracy for the new tasks without having to change the sensor hardware.

We introduce Recurrent State Representation Learning (RSRL) to tackle the problem of state representation learning in robotics for partially observable environments. To learn low-dimensional state representations, we combine a Long Short Term Memory network with robotic priors. RSRL introduces new priors with landmarks and combines them with existing robotics priors from the literature to train the representations. To evaluate the quality of the learned state representation, we introduce validation networks that help us better visualize and quantitatively analyze the learned state representations. We show that the learned representations are low-dimensional, locally consistent, and can approximate the underlying true state for robot localization in simulated 3D maze environments. We use the learned representations for reinforcement learning and show that we achieve similar performance as training with the true state. The learned representations are robust to landmark misclassification errors.

Interactive perception exploits the correlation between forceful interactions and changes in the observed signals to extract task-relevant information from the sensor stream. Finding the most informative interactions to perceive complex objects, like articulated mechanisms, is challenging because the outcome of the interaction is difficult to predict. We propose a method to select the most informative action while deriving a model of articulated mechanisms that includes kinematic, geometric, and dynamic properties. Our method addresses the complexity of the action selection task based on two insights. First, we show that for a class of interactive perception methods, information gain can be approximated by the amount of motion induced in the mechanism. Second, we resort to physics simulations grounded in the real-world through interactive perception to predict possible action outcomes. Our method enables the robot to autonomously select actions for interactive perception that reveal most information, given the current knowledge of the world. This leads to improved perception and more accurate world models, finally enabling robust manipulation.

We demonstrate that moving the wrist while the fingers perform a grasp increases performance. The coordination shapes the interactions between the fingers, the object and its environment to extend the hand capabilities (e.g. higher payload and precision). We evaluated our hypothesis with a human grasping study where the volunteers grasped objects by moving the soft RBO Hand 2 while its fingers closed in a predefined motion. We limited their ability to coordinate their motion with the finger movements using a compliant robot attached to the hand, and observed that their grasp success decreases with increased constraints. We also successfully transferred one of the observed movement patterns to the robot, indicating that adaptive intrinsic/extrinsic motion increases robotic grasp performance as well.

In the context of robotic manipulation and grasping, the shift from a view that is static (force closure of a single posture) and contact-deprived (only contact for force closure is allowed, everything else is obstacle) towards a view that is dynamic and contact-rich (soft manipulation) has led to an increased interest in soft hands. These hands can easily exploit environmental constraints and object surfaces without risk, and safely interact with humans, but present also some challenges. Designing them is difficult, as well as predicting, modelling, and “programming” their interactions with the objects and the environment. This paper tackles the problem of simulating them in a fast and effective way, leveraging on novel and existing simulation technologies. We present a triple-layered simulation framework where dynamic properties such as stiffness are determined from slow but accurate FEM simulation data once, and then condensed into a lumped parameter model that can be used to fast simulate soft fingers and soft hands. We apply our approach to the simulation of soft pneumatic fingers.

A robot with contact sensing capability can reduce uncertainty relative to the environment by deliberately moving into contact and matching the resulting contact measurement to different possible states in the world. We present a manipulation planner that finds and sequences these actions by reasoning explicitly about the uncertainty over the robot's state. The planner incrementally constructs a policy that covers all possible contact states during a manipulation and finds contingencies for each of them. In contrast to conformant planners (without contingencies), the planned contingent policies are more robust. We demonstrate this in simulated and real-world manipulation experiments. In contrast to POMDP-based planners, we show that our planner can be directly applied to high-dimensional configuration spaces.

We propose a novel sensing method for soft pneumatic actuators. The method uses a single microphone, embedded into the actuator's air chamber. Contact with the environment induces sound (vibration) in the actuator. The materials and the shape of the actuator reflect, refract, and attenuate the sound as it propagates inside the actuator. This produces a unique sound signature for different types of events, enabling the sensing of contact locations, contact force, and the type of contacted material. Sensing is insensitive to the inflation state of the actuator and to background noise. We demonstrate the robustness and versatility of the microphone-based sensor solution in experiments with a PneuFlex actuator. The proposed sensorization avoids the fundamental challenges of sensorizing soft pneumatic actuators, because the placement of a microphone does not negatively affect the compliance of the actuator and because a single microphone suffices for sensorization of the entire actuator, eliminating the need for an application-specific sensor layout.

This paper presents an overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning, and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge.

Automated grasping has a long history of research that is increasing due to interest from industry. One grand challenge for robotics is Universal Picking: the ability to robustly grasp a broad variety of objects in diverse environments for applications from warehouses to assembly lines to homes. Although many researchers now openly share code and data, it is challenging to compare and/or reproduce experimental results to identify which aspects of which approaches work best due to variations in assumptions and experimental protocols, e.g., sensors, lighting, robot arms, grippers, and objects.

The most prominent criterion for learning of manipulation skills is the optimization of task success, modeled as expected reward or probability of success. This is sensible if we only want to optimize a single controller. But if learned manipulation primitives are used as modules in a larger system, then it is also important that their generated sensor traces facilitate recognition of action-outcomes. Optimization solely for expected success of a primitive does not guarantee this. We demonstrate a simple example for optimization of actions towards observability, combined with optimization for expected success. Our experiment is a manipulation task with a soft manipulator, where an action primitive is learned such that its generated sensor trace helps a classifier to distinguish task success and task failure. The experimental results indicate that adding auxiliary forces to the original manipulation primitive can indeed facilitate outcome recognition for manipulation tasks.

The compliance of soft actuators makes manipulation safer and simplifies control. But their high flexibility also makes sensorization challenging. From the large space of possible deformations not all are equally important. We present a method for sensorization of soft actuators that, for a given application, finds an effective layout from a set of sensors. It starts from a redundant sensor layout and iteratively reduces the number of sensors. Applying the method to the PneuFlex actuators of the RBO Hand 2, we identify a layout of four liquid metal strain sensors and one pressure sensor to predict actuator deformation in three dimensions: flexional, lateral, and twist. Finally, the layout is used to build a sensorized RBO Hand 2. It can detect passive shape adaptation while grasping and reveals failure cases during manipulation, e.g. slipping fingers while opening a door.

We describe the winning entry to the Amazon Picking Challenge 2015. From the experience of building this system and competing, we derive several conclusions: (1) We suggest to characterize robotic system building along four key aspects, each of them spanning a spectrum of solutions - modularity vs. integration, generality vs. assumptions, computation vs. embodiment, and planning vs. feedback. (2) To understand which region of each spectrum most adequately addresses which robotic problem, we must explore the full spectrum of possible approaches. (3) For manipulation problems in unstructured environments, certain regions of each spectrum match the problem most adequately, and should be exploited further. This is supported by the fact that our solution deviated from the majority of the other challenge entries along each of the spectra. This is an abridged version of a conference publication.

To leverage soft hands to their full potential for grasping, we propose to design their morphology and control signals together. Considering both parameter domains makes it easier and faster to find solutions compared to fixing parameters of either domain. Additionally, the approach scales well to high-dimensional parameter spaces, which is a precondition to make automated co-design useful for soft hands. We further present an efficient simulator for simulating grasps with soft hands which is based on the SOFA framework and enables us to simulate more than a million grasps per day. These two complementary improvements promise a boost in the development of competent soft hands and their control in the future.

Environment-constrained grasping exploits beneficial interactions between hand, object, and environment to increase grasp success. Instead of focusing on the final static relationship between hand posture and object pose, this view of grasping emphasizes the need and the opportunity to select the most appropriate, contact-rich grasping motion, leading up to a final static grasp configuration. This view changes the nature of the underlying planning problem: Instead of planning for static contact points, we need to decide which environmental constraint (EC) to use during the grasping motion. We propose a method to make these decisions based on depth measurements so as to generate robust grasps for a large variety of objects. Our planner exploits the advantages of a soft robot hand and learns a hand-specific classifier for edge-, surface-, and wall-grasps, each exploiting a different EC. Additionally, we show how the model can continuously be improved in a contextual multi-armed bandit setting without an explicit training and test phase, enabling the continuous improvement of a robot's grasping skills throughout life time.

In many robotic applications, softness leads to improved performance, robustness, and safety, while lowering manufacturing cost, increasing versatility, and simplifying control. The advantages of soft robots derive from the fact that their behavior partially results from interactions of the robot's morphology with its environment, which is commonly referred to as morphological computation (MC). But not all MC is good in the sense that it supports the desired behavior. One of the challenges in soft robotics is to build systems that exploit the morphology (good MC) while avoiding body-environment interactions that are harmful with respect to the desired functionality (bad MC). Up to this point, constructing a competent soft robot design requires experience and intuition from the designer. This work is the first to propose a systematic approach that can be used in an automated design process. It is based on calculating a low-dimensional representation of an observed behavior, which can be used to distinguish between good and bad MC. We evaluate our method based on a set of grasping experiments, with variations in hand design, controller, and objects. Finally, we show that the information contained in the low-dimensional representation is comprehensive in the sense that it can be used to guide an automated design process.

Handshakes are common greetings, and humans therefore have strong priors of what a handshake should feel like. This makes it challenging to create compelling and realistic human-robot handshakes, necessitating the consideration of human haptic perception in the design of robot hands. At its most basic level, haptic perception is encoded by contact points and contact pressure distributions on the skin. This motivates our work on measuring the contact area and contact pressure in human handshaking interactions. We present two benchmarking experiments in this regard, measuring the contact locations in human-human/human-robot handshaking and the contact pressure distribution for handshakes with a sensorized palm. We present results from human studies with the benchmarking experiments, providing a baseline for comparison with robot hands as well as presenting new insights into human handshaking. We also show initial work in using these results for the evaluation of robot hands, and progressing towards iterative design of robot hands optimized for social hand interactions.

We present an online system to perceive kinematic properties of articulated objects from multi-modal sensor streams. The novelty of our system is that it leverages multi-modal information in a cross-modal manner: instead of simply fusing information from different modalities, sensor streams are interpreted by leveraging information from another modality. We realize each cross-modal information extraction process using recursive estimation, with each process addressing a perceptual subproblem. Several estimators are then coupled in a cross-modal network, leading to efficient and robust online perception. We demonstrate experimentally that our cross-modal system improves over its uni-modal counterparts, increasing the variability of environments and task conditions in which the robot can robustly perceive the articulated objects. We further demonstrate that these perceptual abilities are sufficiently fast to provide feedback during manipulation actions and sufficiently comprehensive to allow the generation of new manipulation actions.

In this paper we present a planner that interleaves free-space motion with motion in contact to reduce uncertainty. The planner finds such motions by growing a search tree in the combined space of collision-free and contact configurations. The planner reasons efficiently about the accumulated uncertainty by factoring the state in a belief over configuration and a fully observable contact state. We show the uncertainty-reducing capabilities of the planner on manipulation benchmark from the POMDP literature. The planner scales up to more complex problems like manipulation under uncertainty in seven-dimensional configuration space. We validate our planner in simulation and on a real robot.

BACKGROUND: Accurately predicted contacts allow to compute the 3D structure of a protein. Since the solution space of native residue-residue contact pairs is very large, it is necessary to leverage information to identify relevant regions of the solution space, i.e. correct contacts. Every additional source of information can contribute to narrowing down candidate regions. Therefore, recent methods combined evolutionary and sequence-based information as well as evolutionary and physicochemical information. We develop a new contact predictor (EPSILON-CP) that goes beyond current methods by combining evolutionary, physicochemical, and sequence-based information. The problems resulting from the increased dimensionality and complexity of the learning problem are combated with a careful feature analysis, which results in a drastically reduced feature set. The different information sources are combined using deep neural networks. RESULTS: On 21 hard CASP11 FM targets, EPSILON-CP achieves a mean precision of 35.7% for top- L/10 predicted long-range contacts, which is 11% better than the CASP11 winning version of MetaPSICOV. The improvement on 1.5L is 17%. Furthermore, in this study we find that the amino acid composition, a commonly used feature, is rendered ineffective in the context of meta approaches. The size of the refined feature set decreased by 75%, enabling a significant increase in training data for machine learning, contributing significantly to the observed improvements. CONCLUSIONS: Exploiting as much and diverse information as possible is key to accurate contact prediction. Simply merging the information introduces new challenges. Our study suggests that critical feature analysis can improve the performance of contact prediction methods that combine multiple information sources. EPSILON-CP is available as a webservice: http://compbio.robotics.tu-berlin.de/epsilon/.

Recent approaches in robot perception follow the insight that perception is facilitated by interaction with the environment. These approaches are subsumed under the term Interactive Perception (IP). This view of perception provides the following benefits. First, interaction with the environment creates a rich sensory signal that would otherwise not be present. Second, knowledge of the regularity in the combined space of sensory data and action parameters facilitates the prediction and interpretation of the sensory signal. In this survey, we postulate this as a principle for robot perception and collect evidence in its support by analyzing and categorizing existing work in this area. We also provide an overview of the most important applications of IP. We close this survey by discussing remaining open questions. With this survey, we hope to help define the field of Interactive Perception and to provide a valuable resource for future research.

We present an integrated approach for perception of unknown articulated objects. To robustly perceive objects and understand interactions, our method tightly integrates pose tracking, shape reconstruction, and the estimation of their kinematic structure. The key insight of our method is that these sub-problems complement each other: for example, tracking is greatly facilitated by knowing the shape of the object, whereas the shape and the kinematic structure can be more easily reconstructed if the motion of the object is known. Our combined method leverages these synergies to improve the performance of perception. We analyze the proposed method in average cases and difficult scenarios using a variety of rigid and articulated objects. The results show that our integrated solution achieves better results than solutions for the individual problems. This demonstrates the benefits of approaching robot perception problems in an integrated manner.

Soft pneumatic hands offer the advantage of intrinsic mechanical compliance. We argue that to fully leverage the compliance available in soft pneumatic actuators, they should be controlled using air mass rather than position or force, as is customary in most research in soft robotics. We propose an air-mass controller that can servo to a preset position and also allows for the exploitation of fast, mechanical compliance without additional control burden. The proposed mass control scheme is based on discrete commodity valves and pressure sensors, filling a gap in available mass control systems for small-scale soft continuum actuators. The proposed mass controller exhibits low drift for mass trajectories lasting tens of seconds, without requiring a precise model of the actuator. Continuous mass control enables applications for soft robotics, in which leveraging compliance during actuation is of central importance.

The effectiveness of robot interaction depends on the robot's ability to perform task-relevant actions and on the degree to which it is able to predict the outcomes of these actions. In this paper we argue that the two learning problems - learning actions and learning forward models - must be tightly coupled for each of them to be successful. We present an approach that is able to learn a set of continuous action parameters and relational forward models from the robot's own experience. We formalize our approach as simultaneously clustering experiences in a continuous and a relational representation. Our experiments in a simulated manipulation experiment show that this form of coupled subsymbolic and symbolic learning is required for the robot to acquire task-relevant action capabilities.

We present a method for multi-class segmentation from RGB-D data in a realistic warehouse picking setting. The method computes pixel-wise probabilities and combines them to find a coherent object segmentation. It reliably segments objects in cluttered scenarios, even when objects are translucent, reflective, highly deformable, have fuzzy surfaces, or consist of loosely coupled components. The robust performance results from the exploitation of problem structure inherent to the warehouse setting. The proposed method proved its capabilities as part of our winning entry to the 2015 Amazon Picking Challenge. We present a detailed experimental analysis of the contribution of different information sources, compare our method to standard segmentation techniques, and assess possible extensions that further enhance the algorithm's capabilities. We release our software and data sets as open source.

Observations of human grasping reveal that the exploitation of environmental constraints is a key structural aspect for the robustness and versatility of human grasping behavior. We analyze 3,400 human grasping trials with 17 subjects grasping 25 objects to show that viewing environmental constraints as the central structural aspect of human grasping yields surprisingly simple representations of human grasping behavior. We present hypothesis-driven experiments that emphasize the centrality of environmental constraints in human grasping and extract from data a simple “grasping plan” that is a generative model for all of the human grasping trials we observed. This grasping plan can in principle be transferred to a robot system in an attempt to leverage environmental constraints to improve the performance of robotic grasping.

There is strong evidence that robustness in human and robotic grasping can be achieved through the deliberate exploitation of contact with the environment. In contrast to this, traditional grasp planners generally disregard the opportunity to interact with the environment during grasping. In this paper, we propose a novel view of grasp planning that centers on the exploitation of environmental contact. In this view, grasps are sequences of constraint exploitations, i.e. consecutive motions constrained by features in the environment, ending in a grasp. To be able to generate such grasp plans, it becomes necessary to consider planning, perception, and control as tightly integrated components. As a result, each of these components can be simplified while still yielding reliable grasping performance. We propose a first implementation of a grasp planner based on this view and demonstrate in real-world experiments the robustness and versatility of the resulting grasp plans.

As a first step towards transferring human grasping capabilities to robots, we analyzed the grasping behavior of human subjects. We derived a taxonomy in order to adequately represent the observed strategies. During the analysis of the recorded data, this classification scheme helped us to obtain a better understanding of human grasping behavior. We will provide support for our hypothesis that humans exploit compliant contact between the hand and the environment to compensate for uncertainty. We will also show a realization of the resulting grasping strategies on a real robot. It is our belief that the detailed analysis of human grasping behavior will ultimately lead to significant increases in robot manipulation and dexterity.

We present an incremental method for motion generation in environments with unpredictably moving and initially unknown obstacles. The key to the method is its incremental nature: it locally augments and adapts global motion plans in response to changes in the environment, even if they significantly change the connectivity of the world. The restriction to local changes to a global plan results from the fact that in mobile manipulation, robots can ultimately only rely on their on-board sensors to perceive changes in the world. The proposed method addresses three sub-problems of motion generation with three algorithmic components. The first component reactively adapts plans in response to small, continuous changes. The second augments the plan locally in response to connectivity changes. And the third extracts a global, goal-directed motion from the representation maintained by the first two components. In an experimental evaluation of this method, we show a real-world mobile manipulator executing a whole-body motion task in an initially unknown environment, while incrementally maintaining a plan using only on-board sensors.

The ability to selectively stiffen otherwise compliant soft actuators increases their versatility and dexterity. We investigate granular jamming and layer jamming as two possible methods to achieve stiffening with PneuFlex actuators, a type of soft continuum actuator. The paper details five designs of jamming compartments that can be attached to an actuator. We evaluate the stiffening of the five different prototypes, achieving an up to 8-fold increase in stiffness. The strength of the most effective prototype based on layer jamming is also validated in the context of pushing buttons, resulting in an 2.23-fold increase in pushing force.

To successfully manipulate novel objects, robots must first acquire information about the objects' kinematic structure. We present a method for learning relational kinematic background knowledge from exploratory interactions with the world. As the robot gathers experience, this background knowledge enables the acquisition of kinematic world models with increasing efficiency. Learning such background knowledge, however, proves difficult, especially in complex, feature-rich domains. We present a novel, task-sensitive relational rule learner and demonstrate that it is able to learn accurate kinematic background knowledge in domains where other approaches fail. The resulting background knowledge is more compact and generalizes better than that obtained with existing approaches.

Mobile manipulation targets applications in dynamic and unstructured environments. Motion generation methods suitable for these applications must account for end-effector task constraints, must reason about environment uncertainty, i.e. the fact that the exact state of the dynamic environment cannot be known to the robot, and should do so only using their on-board sensors. We present the Expected-Shortest-Path Elastic Roadmap (ESPER) planner as a motion generation method suitable for mobile manipulation. It integrates task-constrained, whole-body, reactive motion generation in high-dimensional configuration space with reasoning about uncertainty. In our experiments, we generate task-consistent motion in uncertain environments on a real-world mobile manipulator only relying on on-board sensors.

We propose a novel method for object search in realistic environments. We formalize object search as a probabilistic inference problem over possible object locations. The method makes two contributions. First, we identify five priors, each capturing structure inherent to the physical world that is relevant to the search problem. Second, we propose a formalization of the object search problem that leverages these priors. Our formalization in form of a probabilistic graphical model is capable of combining the various sources of information into a consistent probability distribution over object locations. The formalization allows us to sharpen the distribution by propagating the knowledge across locations. We employ the reasoning method to select actions of a searching robot in a simulated environment and show that it results in more efficient object search.

To successfully manipulate in unknown environments, a robot must be able to perceive degrees of freedom of objects in its environment. Based on the resulting kinematic model and joint configurations, the robot is able to select and adapt actions, recognize their successful completion and detect failure. We present an RGB-D-based online algorithm for the interactive perception of articulated objects. The algorithm decomposes the perception problem into three interconnected levels of recursive estimation. The estimation problems at each level are much simpler than the original problem and their robustness is improved by level-specific priors that help reject noise in the measurements. These three estimators mutually inform each other to further improve the convergence properties of the three estimation solutions. We demonstrate that the resulting algorithm is robust, accurate, and versatile in real-world experiments. We also show how the perceptual skill can be used online to control the robot's behavior in real-world manipulation tasks.

Depth sensors based on projected structured light have become standard in robotics research. However, when several of these sensors share the same workspace, the measurement quality can deteriorate significantly due to interference of the projected light patterns. We present a comprehensive study of this effect in Kinect and Xtion RGB-D sensors. In particular, our study investigates the effect of measurement failure due to interference. Our experiments show that up to 95% of the depth measurements in the interference image region can disappear when two RGB-D sensors interfere with each other. We determine the severity of interference as a function of relative sensor placement and propose simple guidelines to reduce the impact of sensor interference. We show that these guidelines greatly increase the robustness of RGB-D-based SLAM.

Physical exploration refers to the challenge of autonomously discovering and learning how to manipulate the environment's degrees of freedom (DOF)-by identifying promising points of interaction and pushing or pulling object parts to reveal DOF and their properties. Recent existing work focused on sub-problems like estimating DOF parameters from given data. Here, we address the integrated problem, focusing on the higher-level strategy to iteratively decide on the next exploration point before applying motion generation methods to execute the explorative action and data analysis methods to interpret the feedback. We propose to decide on exploration points based on the expected information gain, or change in entropy in the robot's current belief (uncertain knowledge) about the DOF. To this end, we first define how we represent such a belief. This requires dealing with the fact that the robot initially does not know which random variables (which DOF, and depending on their type, which DOF properties) actually exist. We then propose methods to estimate the expected information gain for an exploratory action. We analyze these strategies in simple environments and evaluate them in combination with full motion planning and data analysis in a physical simulation environment.

We present the exploring/exploiting tree (EET) algorithm for motion planning. The EET planner deliberately trades probabilistic completeness for computational efficiency. This tradeoff enables the EET planner to outperform state-of-the-art sampling-based planners by up to three orders of magnitude. We show that these considerable speedups apply for a variety of challenging real-world motion planning problems. The performance improvements are achieved by leveraging work space information to continuously adjust the sampling behavior of the planner. When the available information captures the planning problem's inherent structure, the planner's sampler becomes increasingly exploitative. When the available information is less accurate, the planner automatically compensates by increasing local configuration space exploration. We show that active balancing of exploration and exploitation based on workspace information can be a key ingredient to enabling highly efficient motion planning in practical scenarios.

The RBO Hand is a novel, highly compliant robotic hand. It exhibits robust grasping performance, is easy to build and prototype, and very cheap to produce. One of the primary design motivations is the extensive leverage of compliance to achieve robust shape matching between the hand and the grasped object. This effect results in robust grasping performance under sensing, model, and actuation uncertainty. We show the feasibility of our approach to constructing robotic hands in extensive grasping experiments on objects with varying properties, included water bottles, eye glasses, and sheets of fabric. The RBO hand is based on a novel pneumatic actuator, called PneuFlex, which exhibits desirable properties for robotic fingers.

In grasping, shape adaptation between hand and object has a major influence on grasp success. In this paper, we present an approach to grasping unknown objects that explicitly considers the effect of shape adaptability to simplify perception. Shape adaptation also occurs between the hand and the environment, for example, when fingers slide across the surface of the table to pick up a small object. Our approach to grasping also considers environmental shape adaptability to select grasps with high probability of success. We validate the proposed shape-adaptability-aware grasping approach in 880 real-world grasping trials with 30 objects. Our experiments show that the explicit consideration of shape adaptability of the hand leads to robust grasping of unknown objects. Simple perception suffices to achieve this robust grasping behavior.

To perform successful manipulation, robots depend on information about objects in their environment. In unstructured environments, such information cannot be given to the robot a priori. It is thus critical for the robot to be able to continuously acquire task-specific information about objects. Towards this goal, we present a robust perceptual skill for identifying, tracking, and segmenting objects in a cluttered environment. We increase the robot's perceptual capabilities by closely coupling them with the robot's manipulation skills. The robot's interaction with objects in the environment creates a perceptual signal, i.e. motion, that renders segmentation and tracking robust and reliable. In addition, the resulting perceptual signal reveals the type of segmentation most relevant to manipulation, namely a segmentation of rigidly connected physical bodies. We demonstrate our approach with experiments on a real world mobile manipulation platform with multiple objects in a cluttered scene.

Robust robotic manipulation and perception remains a difficult challenge, in particular in unstructured environments. To address this challenge, we propose to couple manipulation and perception. The robot observes its own deliberate interactions with the world. These interactions reveal sensory information that would otherwise remain hidden and facilitate the interpretation of perceptual data. To demonstrate the effectiveness of interactive perception we present a skill for the manipulation of articulated objects. We show how UMan, our mobile manipulation platform, obtains a kinematic model of an unknown object. The model then enables the robot to perform purposeful manipulation. Our algorithm is extremely robust, and does not require prior knowledge of the object; it is insensitive to lighting, texture, color, specularities, background, and is computationally highly efficient.

Computationally efficient motion planning must avoid exhaustive exploration of configuration space. We argue that this can be accomplished most effectively by carefully balancing exploration and exploitation. Exploration seeks to understand configuration space, irrespective of the planning problem, while exploitation acts to solve the problem given the available information obtained by exploration. We present an exploring/exploiting tree (EET) planner that balances its exploration and exploitation behavior. The planner acquires workspace information and subsequently uses this information for exploitation in configuration space. If exploitation fails in difficult regions, the planner gradually shifts its behavior towards exploration. We present experimental results demonstrating that adaptive balancing of exploration and exploitation leads to significant performance improvements compared to other state-of-the-art sampling-based planners.

Randomly expanding trees are very effective in exploring high-dimensional spaces. Consequently, they are a powerful algorithmic approach to sampling-based single-query motion planning. As the dimensionality of the configuration space increases, however, the performance of tree-based planners that use uniform expansion degrades. To address this challenge, we present a utility-guided algorithm for the online adaptation of the random tree expansion strategy. This algorithm guides expansion towards regions of maximum utility based on local characteristics of state space. To guide exploration, the algorithm adjusts the parameters that control random tree expansion in response to state space information obtained during the planning process. We present experimental results to demonstrate that the resulting single-query planner is computationally more efficient and more robust than previous planners in challenging artificial and real-world environments.

Sampling-based algorithms have dramatically improved the state of the art in robotic motion planning. However, they make restrictive assumptions that limit their applicability to manipulators operating in uncontrolled and partially unknown environments. This work describes how one of these assumptions - that the world is perfectly known - can be removed. We propose a utility-guided roadmap planner that incorporates uncertainty directly into the planning process. This enables the planner to identify configuration space paths that minimize uncertainty and, when necessary, efficiently pursue further exploration through utility-guided sensing of the workspace. Experimental results indicate that our utility-guided approach results in a robust planner even in the presence of significant error in its perception of the workspace. Furthermore, we show how the planner is able to reduce the amount of required sensing to compute a successful plan

Mobile robots have successfully solved many real world problems. In the following we present the use of mobile robots to address the novel and challenging problem of providing disruption tolerant network service. In disruption tolerant networks, all messages are transported by the physical motion of participants in the network. When these movements do not meet the service demands of the network, network performance can only be improved by adding robots that provide additional network service. The task of controlling such robots is a problem that is NP-hard. To develop an approximate solution, we propose a nullspace-based algorithm for controlling the motion of the added robots. This controller simultaneously optimizes multiple network performance metrics. Experiments that simulate the addition of robots to a real-world disruption tolerant network show that the introduction of mobile robots running our control scheme can significantly improve the performance and service guarantees of a disruption tolerant network

Efficient motion planning for robots with many degrees of freedom requires the exploration of a large configuration space. Sampling based motion planners perform approximate exploration of the configuration space in order to render the problem tractable. Each sample of configuration space as an opportunity to gain information about that configuration space. A formal definition of information gain can be used to guide a motion planner to achieve maximal progress toward the discovery of a path. We call such a motion planner entropy-guided since entropy reduction is synonymous with information gain. In the following we describe a single-query entropy-guided motion planner which uses a formal definition of information gain to focus its efforts on the acquisition of a single path from start to goal locations. Experimental evidence indicates that this approach can outperform existing single-query techniques.

Robotic motion planning requires configuration space exploration. In high-dimensional configuration spaces, a complete exploration is computationally intractable. Practical motion planning algorithms for such high-dimensional spaces must expend computational resources in proportion to the local complexity of configuration space regions. We propose a novel motion planning approach that addresses this problem by building an incremental, approximate model of configuration space. The information contained in this model is used to direct computational resources to difficult regions, effectively addressing the narrow passage problem by adapting the sampling density to the complexity of that region. In addition, the expressiveness of the model permits predictive edge validations, which are performed based on the information contained in the model rather then by invoking a collision checker. Experimental results show that the exploitation of the information obtained through sampling and represented in a predictive model results in a significant decrease in the computational cost of motion planning.

Disruption-tolerant networks (DTNs) differ from other types of networks in that capacity is exclusively created by the movements of participants. This implies that understanding and influencing the participants' motions can have a significant impact on network performance. In this paper, we introduce the routing protocol MV, which learns structure in the movement patterns of network participants and uses it to enable informed message passing. We also propose the introduction of autonomous agents as additional participants in DTNs. These agents adapt their movements in response to variations in network capacity and demand. We use multi-objective control methods from robotics to generate motions capable of optimizing multiple network performance metrics simultaneously. We present experimental evidence that these strategies, individually and in conjunction, result in significant performance improvements in DTNs.

In this paper we propose a new approach for multi-objective control using a cascade of filters that progressively removes candidate commands which do not satisfy task constraints. The approach is motivated by other control methods that prevent destructive control interactions through null space projections. We apply this approach to a practical leader/follower task in which a mobile robot must address the conflicting objectives of moving to a goal position while avoiding obstacles and keeping a region of the workspace within the field of view of a fixed camera mounted on the platform. We experimentally verify our approach using the Segway Robotic Mobility Platform (RMP), a dynamically stable, differential drive mobile robot.

Probabilistic roadmap planners have proven to be effective in solving complex path planning problems. These planners sample the configuration space to compute a representation of its free space connectivity. One of the major difficulties for this approach is the planning of a path through narrow configuration space passages, since samples are placed inside narrow passages only with small probability. To address this problem, approaches have been devised that rely on the medial axis of the workspace to bias sampling in configuration space such that the probability of generating samples inside narrow passages is increased. This paper introduces a novel algorithm for computing an approximation to the medial axis, which can be computed more efficiently than the exact or discretized medial axis. We demonstrate that, compared to the true medial axis, this approximation is equally well suited to bias the sampling in probabilistic roadmap planners. Furthermore, we present a novel sampling strategy based on the approximated medial axis. This strategy results in high sampling density in narrow passages, while sampling open spaces sparsely. Experiments demonstrate the effectiveness of the medial axis approximation and its application to motion planning based on the proposed sampling scheme.

Teams of robots can be redundant with respect to a given task. This redundancy can be exploited to pursue additional objectives during the execution of the task. In this paper, we describe a control-based method to exploit such redundancy for the execution of additional behavior, leading to the improvement of overall performance. The control-based method provides a suitable mechanism for combining controllers with different objectives. The mechanism ensures that the subordinate controllers do not interfere with the superior controllers. Thus it allows to build controllers exhibiting complex behavior from simple primitives, while maintaining their provable performance characteristics. The effectiveness of the framework is demonstrated by experiments with a multi-robot exploration task.

Probabilistic roadmaps (PRM) are a randomized tool for path planning in configuration spaces where exhaustive search is computationally intractable. It has been noted that the PRM algorithm's computational cost can be greatly reduced by reducing the number of samples necessary to construct a successful roadmap. We examine the information theoretic properties of roadmap construction and propose sampling techniques based upon maximizing the information gain of the roadmap for each configuration sampled. Instead of sampling algorithms which are meant to understand the entirety of configuration space, our sampling is focused on finding configurations which facilitate roadmap construction. We show empirically that these approaches can lead to a significant reduction in the number of samples necessary to construct a useful roadmap.

Applications in mobile manipulation require sophisticated motion execution skills to address issues like redundancy resolution, reactive obstacle avoidance, and transitioning between different motion behaviors. The elastic strip framework is an approach to reactive motion generation providing an integrated solution to these problems. Novel techniques within the elastic strip framework are presented, allowing task-consistent obstacle avoidance and task-consistent motion behavior. General transition criteria and methods are presented, permitting the suspension and resumption of task execution to ensure other desired motion behavior, such as obstacle avoidance. Task execution has to be suspended when kinematic constraints or changes in the environment render task-consistent motion behavior infeasible. Task execution is resumed as soon as it is consistent with other desired motion behaviors.

Research in motion planning has been striving to develop faster planning algorithms in order to be able to address a wider range of applications. In this paper a novel real-time motion planning framework, called decomposition-based motion planning, is proposed. It is particularly well suited for planning problems that arise in service and field robotics. It decomposes the original planning problem into simpler sub-problems, whose successive solution empirically results in a large reduction of the overall complexity. A particular implementation of decomposition-based planning is proposed. Experiments with an eleven degree-of-freedom mobile manipulator are presented.

Real-time replanning is a prerequisite for motion execution in unpredictably changing environments. This paper presents a framework that allows real-time replanning in high-dimensional configuration spaces. Initially, a planning operation generates a path. The path is augmented by a set of paths homotopic to it. This set is represented implicitly by a volume of free space in the work space. Effectively, this corresponds to delaying part of the planning operation for the homotopic paths until motion execution. During execution reactive control algorithms are used to select a valid path from the set of homotopic paths, using proximity to the environment as a simple and effective heuristic and thereby significantly pruning the search in the configuration space. Experimental results are presented to validate the real-time performance of this framework in high-dimensional configuration spaces.

Many applications in mobile robotics require the safe execution of a collision-free motion to a goal position. Planning approaches are well suited for achieving a goal position in known static environments, while real-time obstacle avoidance methods allow reactive motion behavior in dynamic and unknown environments. This paper proposes the global dynamic window approach as a generalization of the dynamic window approach. It combines methods from motion planning and real-time obstacle avoidance to result in a framework that allows robust execution of high-velocity, goal-directed reactive motion for a mobile robot in unknown and dynamic environments. The global dynamic window approach is applicable to nonholonomic and holonomic mobile robots.

In many robotic applications motions must be executed robustly in dynamic and partially unknown environments. Despite this requirement most motion planning algorithms assume the environment to be known and changes to be predictable. The planning problem in dynamic environments can be decomposed into a planning and an execution phase. In this paper we describe a new framework for the execution of motion plans for robots with many degrees of freedom in dynamic environments. An initial valid trajectory is incrementally modified according to changes in the environment to maintain a collision free path. This framework achieves real-time performance for robots with many degrees of freedom. It is particularly well suited for redundant systems and mobile manipulation, since it allows motion specification of a subset of the degrees of freedom of the robot, greatly simplifying the task of robot programming.