The morphology and controller design of robots is often a labor-intensive task performed by experienced and intuitive engineers. Automatic robot design using machine learning is attracting increasing attention in the hope that it will reduce the design workload and result in better-performing robots. Most robots are created by joining several rigid parts and then mounting actuators and their controllers. Many studies limit the possible types of rigid parts to a finite set to reduce the computational burden. However, this not only limits the search space, but also prohibits the use of powerful optimization techniques. To find a robot closer to the global optimal design, a method that explores a richer set of robots is desirable. In this article, we propose a novel method to efficiently search for various robot designs. The method combines three different optimization methods with different characteristics. We apply proximal policy optimization (PPO) or soft actor-critic (SAC) as the controller, the REINFORCE algorithm to determine the lengths and other numerical parameters of the rigid parts, and a newly proposed method to determine the number and layout of the rigid parts and joints. Experiments with physical simulations confirm that when this method is used to handle two types of tasks-walking and manipulation-it performs better than simple combinations of existing methods. The source code and videos of our experiments are available online (https://github.com/r-koike/eagent).

In this article, we propose a novel variant of path integral policy improvement with covariance matrix adaptation (PI2-CMA), which is a reinforcement learning (RL) algorithm that aims to optimize a parameterized policy for the continuous behavior of robots. PI2-CMA has a hyperparameter called the temperature parameter, and its value is critical for performance; however, little research has been conducted on it and the existing method still contains a tunable parameter, which can be critical to performance. Therefore, tuning by trial and error is necessary in the existing method. Moreover, we show that there is a problem setting that cannot be learned by the existing method. The proposed method solves both problems by automatically adjusting the temperature parameter for each update. We confirmed the effectiveness of the proposed method using numerical tests.

This article describes a novel sufficient condition concerning approximations with reservoir computing (RC). Recently, RC using a physical system as the reservoir has attracted attention. Because many physical systems are modeled as state-space systems, it is necessary to guarantee the approximations given by reservoirs represented as nonlinear state-space systems. There are two problems with existing approaches: a reservoir must have a property called fading memory and must be represented as a set of maps between input and output signals on the bi-infinite-time (BIT) interval. These two conditions are too strict for reservoirs represented as nonlinear state-space systems as they require the reservoir to have a unique equilibrium state for the zero input. This article proposes an approach that employs operators from right-infinite-time (RIT) inputs to RIT outputs. Furthermore, we develop a novel extension of the Stone-Weierstrass theorem to handle discontinuous functions. To apply the extended theorem, we define functionals corresponding to operators and introduce a metric on the domain of the functionals. The resulting sufficient condition does not require the reservoir to have fading memory or continuity with respect to inputs and time. Therefore, our result guarantees the approximations with very common reservoirs and provides a rationale for physical RC. We present an example of a physical reservoir without fading memory. With the example reservoir, the RC model successfully approximates NARMA10, a benchmark task for time series predictions.

Path integral policy improvement (PI2) is known to be an efficient reinforcement learning algorithm, particularly, if the target system is a high-dimensional dynamical system. However, PI2, and its existing extensions, have adjustable parameters, on which the efficiency depends significantly. This article proposes an extension of PI2that adjusts all of the critical parameters automatically. Motion acquisition tasks for three different types of simulated legged robots were performed to test the efficacy of the proposed algorithm. The results show that the proposed method cannot only eliminate the burden on the user to set the parameters appropriately but also improve the optimization performance significantly. For one of the acquired motions, a real robot experiment was conducted to show the validity of the motion.

In the present paper, a dynamic analysis is presented, comparing three snake-like robot gaits: lateral undulation, sidewinding locomotion, and sinus-lifting motion. To simplify calculations, sidewinding locomotion and sinus-lifting motion are considered planar movements. Vertical movements are assumed to be small but play a critical role in change where contacts are made. Thus, the normal forces acting on grounded links and the torques applied to pitch joints can be calculated by solving equilibrium equations. The tradeoff between locomotion speed and energy efficiency is studied for all three gaits, at eight different environmental settings distinguished by friction coefficients. Simulation results reveal that sinus-lifting motion and sidewinding locomotion are generally more energy-efficient gaits than is lateral undulation. More specifically, if the anisotropy in friction is large enough, sinus-lifting motion is the most energy-efficient gait; otherwise, sidewinding locomotion is more efficient. However, there are some critical speeds at which the most efficient gait changes, in some environmental settings.

In many engineering problems, including those related to robotics, optimization of the control policy for multiple conflicting criteria is required. However, this can be very challenging because of the existence of noise, which may be input dependent or heteroscedastic, and restrictions regarding the number of evaluations owing to the costliness of the experiments in terms of time and/or money. This paper presents a multiobjective optimization algorithm for expensive-to-evaluate noisy functions for robotics. We present a method for model selection between heteroscedastic and standard homoscedastic Gaussian process regression techniques to create suitable surrogate functions from noisy samples, and to find the point to be observed at the next step. This algorithm is compared against an existing multiobjective optimization algorithm, and then used to optimize the speed and head stability of the sidewinding gait of a snake robot.

In many robotic problems, optimization of the policy for multiple conflicting criteria is required. However this is very challenging due to the existence of noise, which may be input dependent, or heteroscedastic, and the restriction in the number of evaluations, due to robotic experiments which are expensive in time and/or money. This paper presents a multiobjective optimization (MOO) algorithm for expensive-to-evaluate noisy functions for robotics. We present a method for model selection between heteroscedastic and standard homoscedastic Gaussian process regression techniques to create suitable surrogate functions from noisy samples and find the point to be observed at the next step. This algorithm is compared against an existing MOO algorithm which assumes homoscedastic noise, and is then used to optimize the speed and head stability of the sidewinding gait of a snake robot.

In this paper, a dynamical analysis of sidewinding locomotion by a snake like robot is presented. Sidewinding locomotion is treated as a planar movement for simplicity, and the normal forces acting on the grounded links and the torques applied on pitch joints are calculated by solving equilibrium equations. Energy efficiency of sidewinding locomotion is compared with that of lateral undulation. Simulation results show that the sidewinding locomotion is, generally, energetically more efficient, and in the case where the viscous coefficient is small, there is a lower bound in the mean velocity of center of mass for sidewinding locomotion to be more efficient than lateral undulation.

Central pattern generators (CPGs) play a crucial role for animal locomotion control. They can be entrained by sensory feedback to induce proper rhythmic patterns and even store the entrained patterns through connection weights. Inspired by this biological finding, we use four adaptive neural oscillators with synaptic plasticity as CPGs for locomotion control of our real snake-like robot with screw-drive mechanism. Each oscillator consists of only three neurons and uses adaptive mechanisms based on frequency adaptation and Hebbian-type learning rules. It autonomously generates proper periodic patterns for the robot locomotion and can be entrained by sensory feedback to memorize the patterns. The adaptive CPG system in conjunction with a simple control strategy enables the robot to perform self-tuning behavior which is robust against short-time perturbations. The generated behavior is also energy efficient. In addition, the robot can also cope with corners as well as move through a complex environment with obstacles.

To achieve a balanced walking for a humanoid, it is necessary to estimate the dynamic stability of the system. However, in a small size humanoid with restricted system resource, it is hard to satisfy the performance level desired by dynamics analysis. Therefore, in this paper, we propose the feasible methods to generate gait pattern and stabilize walking based on coupled oscillators which have a clear correlation between oscillator parameters and system dynamics without a real time ZMP calculation. The proposed method was tested on the open humanoid platform DARwIn-OP for the evaluation, and the result showed that a real time gait pattern generation and stabilization were realized.