Reinforcement learning (RL) methods based on direct policy search (DPS) have been actively discussed to achieve an efficient approach to complicated Markov decision processes (MDPs). Although they have brought much progress in practical applications of RL, there still remains an unsolved problem in DPS related to model selection for the policy. In this paper, we propose a novel DPS method, {\it weighted likelihood policy search (WLPS)}, where a policy is efficiently learned through the weighted likelihood estimation. WLPS naturally connects DPS to the statistical inference problem and thus various sophisticated techniques in statistics can be applied to DPS problems directly. Hence, by following the idea of the {\it information criterion}, we develop a new measurement for model comparison in DPS based on the weighted log-likelihood.

Elevator group supervisory control system (EGSCS) is a traffic system, where its controller manages the elevator movement to transport passengers in buildings efficiently. Recently, artificial intelligence (AI) technology has been used in such complex systems. Genetic network programming (GNP), a graph-based evolutionary method extended from GA and GP, has been already applied to EGSCS. On the other hand, since energy consumption is becoming one of the greatest challenges in the society, it should be taken as criteria of the elevator operations. Moreover, the elevator with maximum energy efficiency is therefore required. Finally, the simulations show that the elevator system has the higher energy consumption in the light traffic, thus, some factors have been introduced into GNP for energy saving in this paper.

Reinforcement learning (RL) methods based on least-squares temporal difference (LSTD) have been developed recently and have shown good practical performance. However, the quality of their estimation has not been well elucidated. In this article, we discuss LSTD-based policy evaluation from the new view-point of semiparametric statistical inference. In fact, the estimator can be obtained from a particular estimating function which guarantees its convergence to the true value asymptotically, without specifying a model of the environment. Based on these observations, we 1) analyze the asymptotic variance of an LSTD-based estimator, 2) derive the optimal estimating function with the minimum asymptotic estimation variance, and 3) derive a suboptimal estimator to reduce the computational burden in obtaining the optimal estimating function.

Recently, many researchers on humanoid robotics are interested in quasi-passive-dynamic walking (quasi-PDW) which is similar to human walking. It is desirable that control parameters in quasi-PDW are automatically adjusted because robots often suffer from changes in their physical parameters and the surrounding environment. Reinforcement learning (RL) can be a key technology to this adaptability, and it has been shown that RL realizes quasi-PDW in a simulation study. To apply the existing method to controlling real robots, however, requires further improvement to accelerate its learning, otherwise the robots will break down before acquiring appropriate controls. To accelerate the learning, this study employs off-policy natural actor-critic (off-NAC), and applies it to an acquisition problem of quasi-PDW. The most important feature of the off-NAC is that it reuses the samples that has already been obtained by previous controllers. This study also shows an adaptive method of the learning rate. Simulation as well as real experiments demonstrate that fast and stable learning of quasi-PDW of an unstable biped robot can be realized by our modified off-NAC