Pseudospectral collocation methods have proven to be powerful tools to solve optimal control problems. While these methods generally assume the dynamics is given in the first order form$x$= f(x, u, t), where$x$is the state and$u$is the control vector, robotic systems are typically governed by second order ODEs of the form$q$= g(q, q, u, t), where$q$is the configuration. To convert the second order ODE into a first order one, the usual approach is to introduce a velocity variable$v$and impose its coincidence with the time derivative of q. Lobatto methods grant this constraint by construction, as their polynomials describing the trajectory for$v$are the time derivatives of those for q, but the same cannot be said for the Gauss and Radau methods. This is problematic for such methods, as then they cannot guarantee that$q$= g(q, q, u, t) at the collocation points. On their negative side, Lobatto methods cannot be used to solve initial value problems, as given the values of$u$at the collocation points they generate an overconstrained system of equations for the states. In this paper, we propose a Legendre-Gauss collocation method that retains the advantages of the usual Lobatto, Gauss, and Radau methods, while avoiding their shortcomings. The collocation scheme we propose is applicable to solve initial value problems, preserves the consistency between the polynomials for$v$and q, and ensures that$q$= g(q, q, u, t) at the collocation points.

Function approximation in online, incremental, reinforcement learning needs to deal with two fundamental problems: biased sampling and nonstationarity. In this kind of task, biased sampling occurs because samples are obtained from specific trajectories dictated by the dynamics of the environment and are usually concentrated in particular convergence regions, which in the long term tend to dominate the approximation in the less sampled regions. The nonstationarity comes from the recursive nature of the estimations typical of temporal difference methods. This nonstationarity has a local profile, varying not only along the learning process but also along different regions of the state space. We propose to deal with these problems using an estimation of the probability density of samples represented with a gaussian mixture model. To deal with the nonstationarity problem, we use the common approach of introducing a forgetting factor in the updating formula. However, instead of using the same forgetting factor for the whole domain, we make it dependent on the local density of samples, which we use to estimate the nonstationarity of the function at any given input point. To address the biased sampling problem, the forgetting factor applied to each mixture component is modulated according to the new information provided in the updating, rather than forgetting depending only on time, thus avoiding undesired distortions of the approximation in less sampled regions.

In the online version of the EM algorithm introduced by Sato and Ishii ( 2000 ), a time-dependent discount factor is introduced for forgetting the effect of the old estimated values obtained with an earlier, inaccurate estimator. In their approach, forgetting is uniformly applied to the estimators of each mixture component depending exclusively on time, irrespective of the weight attributed to each unit for the observed sample. This causes an excessive forgetting in the less frequently sampled regions. To address this problem, we propose a modification of the algorithm that involves a weight-dependent forgetting, different for each mixture component, in which old observations are forgotten according to the actual weight of the new samples used to replace older values. A comparison of the time-dependent versus the weight-dependent approach shows that the latter improves the accuracy of the approximation and exhibits much greater stability.

Recent approaches to Reinforcement Learning (RL) with function approximation include Neural Fitted Q Iteration and the use of Gaussian Processes. They belong to the class of fitted value iteration algorithms, which use a set of support points to fit the value-function in a batch iterative process. These techniques make efficient use of a reduced number of samples by reusing them as needed, and are appropriate for applications where the cost of experiencing a new sample is higher than storing and reusing it, but this is at the expense of increasing the computational effort, since these algorithms are not incremental. On the other hand, non-parametric models for function approximation, like Gaussian Processes, are preferred against parametric ones, due to their greater flexibility. A further advantage of using Gaussian Processes for function approximation is that they allow to quantify the uncertainty of the estimation at each point. In this paper, we propose a new approach for RL in continuous domains based on Probability Density Estimations. Our method combines the best features of the previous methods: it is non-parametric and provides an estimation of the variance of the approximated function at any point of the domain. In addition, our method is simple, incremental, and computationally efficient. All these features make this approach more appealing than Gaussian Processes and fitted value iteration algorithms in general.

A central problem in reinforcement learning is how to deal with large state and action spaces. When the problem domain presents intrinsic symmetries, exploiting them can be key to achieve good performance. We analyze the gains that can be effectively achieved by exploiting different kinds of symmetries, and the effect of combining them, in a test case: the stand-up and stabilization of an inverted pendulum.

This paper presents a relational positioning methodology that allows to restrict totally or partially the movements of an object by specifying its allowed positions in terms of a set of intuitive geometric constraints. In order to derive these positions, ageometric constraint solvermust be used. To this end, positioning mobile with respect to fixed (PMF), a geometric constraint solver for the relational positioning of rigid objects in free space is introduced. The solver exploits the fact that, in a set of geometric constraints, the rotational component can often be separated from the translational one and solved independently. PMF may be used as an interface for specifying offline-programmed robot tasks, as well as for assisting the execution of teleoperated tasks requiring constrained movements. Examples describing both the solver's operation and typical applications are discussed.

In this paper, we confront the problem of applying reinforcement learning to agents that perceive the environment through many sensors and that can perform parallel actions using many actuators as is the case in complex autonomous robots. We argue that reinforcement learning can only be successfully applied to this case if strong assumptions are made on the characteristics of the environment in which the learning is performed, so that the relevant sensor readings and motor commands can be readily identified. The introduction of such assumptions leads to strongly-biased learning systems that can eventually lose the generality of traditional reinforcement-learning algorithms. In this line, we observe that, in realistic situations, the reward received by the robot depends only on a reduced subset of all the executed actions and that only a reduced subset of the sensor inputs (possibly different in each situation and for each action) are relevant to predict the reward. We formalize this property in the so called 'categorizability assumption' and we present an algorithm that takes advantage of the categorizability of the environment, allowing a decrease in the learning time with respect to existing reinforcement-learning algorithms. Results of the application of the algorithm to a couple of simulated realistic-robotic problems (landmark-based navigation and the six-legged robot gait generation) are reported to validate our approach and to compare it to existing flat and generalization-based reinforcement-learning approaches.

Real world robot applications have to cope with large variations in the operating conditions due to the variability and unpredictability of the environment and its interaction with the robot. Performing an adequate control using conventional control techniques, that require the model of the plant and some knowledge about the influence of the environment, could be almost impossible. An alternative to traditional control techniques is to use an automatic learning system that uses previous experience to learn an adequate control policy. Learning by experience has been formalized in the field of reinforcement learning. But the application of reinforcement learning techniques in complex environments is only feasible when some generalization can be made in order to reduce the required amount of experience. This work presents an algorithm that performs a kind of generalization called categorization. This algorithm is able to perform efficient generalization of the observed situations, and learn accurate control policies in a short time without any previous knowledge of the plant and without the need of any kind of traditional control technique. Its performance is evaluated on the trajectory tracking control with simulated DC motors and compared with PID systems specifically tuned for the same problem.

Legged robots are well suited to walk on difficult terrains at the expense of requiring complex control systems to walk even on flat surfaces. But simply walking on a flat surface is not worth using a legged robot. It should be assumed that walking on abrupt terrain is the typical situation for a legged robot. With this premise in mind, we have developed a robust controller for a six-legged robot that allows it to walk over difficult terrains in an autonomous way, with a limited use of sensory information (no vision is involved). This walk controller can be driven by an upper level which need not be concerned about the details of foot placement or leg movements, taking care only of high level aspects such as global speed and direction.