To successfully manipulate novel objects, robots must first acquire information about the objects' kinematic structure. We present a method for learning relational kinematic background knowledge from exploratory interactions with the world. As the robot gathers experience, this background knowledge enables the acquisition of kinematic world models with increasing efficiency. Learning such background knowledge, however, proves difficult, especially in complex, feature-rich domains. We present a novel, task-sensitive relational rule learner and demonstrate that it is able to learn accurate kinematic background knowledge in domains where other approaches fail. The resulting background knowledge is more compact and generalizes better than that obtained with existing approaches.

Formal exploration approaches in model-based reinforcement learning estimate the accuracy of the currently learned model without consideration of the empirical prediction error. For example, PAC-MDP approaches such as Rmax base their model certainty on the amount of collected data, while Bayesian approaches assume a prior over the transition dynamics. We propose extensions to such approaches which drive exploration solely based on empirical estimates of the learner's accuracy and learning progress. We provide a ``sanity check'' theoretical analysis, discussing the behavior of our extensions in the standard stationary finite state-action case. We then provide experimental studies demonstrating the robustness of these exploration measures in cases of non-stationary environments or where original approaches are misled by wrong domain assumptions.

A new approach to planning and goal-directed behavior has recently been proposed using probabilistic inference in a graphical model that represents states, actions, constraints and goals of the future to infer appropriate actions and controls. The approach has led to new algorithms on the control and trajectory optimization level as well as for high-level rule-based planning in relational domains. In this paper we integrate these methods to a coherent control, trajectory optimization, and action planning architecture, using the principle of planning by inference across all levels of abstractions. Our scenario is a real blocks world: using a 14DoF Schunk arm and hand with tactile sensors and a stereo camera, the goal is to manipulate a set of objects on the table in a goal-oriented way. For high-level reasoning, we learn relational rule-based models from experience in simulation.

Noisy probabilistic relational rules are a promising world model representation for several reasons. They are compact and generalize over world instantiations. They are usually interpretable and they can be learned effectively from the action experiences in complex worlds. We investigate reasoning with such rules in grounded relational domains. Our algorithms exploit the compactness of rules for efficient and flexible decision-theoretic planning. As a first approach, we combine these rules with the Upper Confidence Bounds applied to Trees (UCT) algorithm based on look-ahead trees. Our second approach converts these rules into a structured dynamic Bayesian network representation and predicts the effects of action sequences using approximate inference and beliefs over world states. We evaluate the effectiveness of our approaches for planning in a simulated complex 3D robot manipulation scenario with an articulated manipulator and realistic physics and in domains of the probabilistic planning competition. Empirical results show that our methods can solve problems where existing methods fail.

Relational world models that can be learned from experience in stochastic domains have received significant attention recently. However, efficient planning using these models remains a major issue. We propose to convert learned noisy probabilistic relational rules into a structured dynamic Bayesian network representation. Predicting the effects of action sequences using approximate inference allows for planning in complex worlds. We evaluate the effectiveness of our approach for online planning in a 3D simulated blocksworld with an articulated manipulator and realistic physics. Empirical results show that our method can solve problems where existing methods fail.