Legged robots have excellent terrestrial mobility for traversing diverse environments and, thus, have the potential to be deployed in a wide variety of scenarios. However, they are susceptible to falling and leg malfunction during locomotion. Although the use of a large number of legs can overcome these problems, it makes the body long and leads to many legs being constrained to contact with the ground to support the long body, which impedes maneuverability. To improve the locomotion maneuverability of such robots, this study focuses on dynamic instability, which induces rapid and large movement changes, and uses a 12-legged robot with a flexible body axis. Our previous work found that the straight walk of the robot becomes unstable through a Hopf bifurcation when the body-axis flexibility is changed, which induces body undulations. Furthermore, we developed a simple controller based on the Hopf bifurcation and showed that the instability facilitates the turning of the robot. In this study, we newly found that the straight walk becomes unstable through a pitchfork bifurcation when the body-axis flexibility is changed in a way different from that in our previous work. In addition, the pitchfork bifurcation induces a transition into a curved walk, whose curvature can be controlled by the body-axis flexibility. We developed a simple controller based on the pitchfork bifurcation characteristics and demonstrated that the robot can perform a turning maneuver superior to that with the previous controller. This study provides a novel design principle for maneuverable locomotion of many-legged robots using intrinsic dynamic properties.

This paper discusses model estimation in offline model-based reinforcement learning (MBRL), which is important for subsequent policy improvement using an estimated model. From the viewpoint of covariate shift, a natural idea is model estimation weighted by the ratio of the state-action distributions of offline data and real future data. However, estimating such a natural weight is one of the main challenges for off-policy evaluation, which is not easy to use. As an artificial alternative, this paper considers weighting with the state-action distribution ratio of offline data and simulated future data, which can be estimated relatively easily by standard density ratio estimation techniques for supervised learning. Based on the artificial weight, this paper defines a loss function for offline MBRL and presents an algorithm to optimize it. Weighting with the artificial weight is justified as evaluating an upper bound of the policy evaluation error. Numerical experiments demonstrate the effectiveness of weighting with the artificial weight.

Passive dynamic walking is a model that walks down a shallow slope without any control or input. This model has been widely used to investigate how stable walking is generated from a dynamic viewpoint, which is useful to provide design principles for developing energy-efficient biped robots. However, the basin of attraction is very small and thin, and it has a fractal-like complicated shape. This makes it difficult to produce stable walking. Furthermore, the passive dynamic walking shows chaotic attractor through a period-doubling cascade by increasing the slope angle, and the chaotic attractor suddenly disappears at a critical slope angle. These make it further difficult to produce stable walking. In our previous work, we used the simplest walking model and investigated the fractal-like basin of attraction based on dynamical systems theory by focusing on the hybrid dynamics of the model composed of the continuous dynamics with saddle hyperbolicity and the discontinuous dynamics by the impact at foot contact. We elucidated that the fractal-like basin of attraction is generated through iterative stretch and bending deformations of the domain of the Poincaré map by sequential inverse images of the Poincaré map. In this study, we investigated the mechanism for the disappearance of the chaotic attractor by improving our previous analysis. In particular, we focused on the range of the Poincaré map to specify the regions to be stretched and bent by the inverse image of the Poincaré map. We clarified the condition for the chaotic attractor to disappear and the mechanism why the chaotic attractor disappears based on the stretch-bending deformation in the basin of attraction.

In this paper, we proposed a control system with learning to investigate neuromechanical functions for generating adaptive bipedal locomotion on a splitbelt treadmill. Humans show two types of adaptations, called early adaptation and late adaptation, in splitbelt treadmill walking. In our previous work, we investigated the locomotor behavior of a biped robot driven by nonlinear oscillators with phase resetting and showed that it produced the early adaptation like humans. However, because the locomotion control system did not contain any learning mechanism, it did not show the late adaptation. In this paper, we newly develop learning systems, which modulate the interlimb and intralimb coordination patterns, and incorporate them to our locomotion control system. We investigated the locomotor behavior of a biped robot using computer simulations and robot experiments. The results showed the early and late adaptations during the splitbelt treadmill walking and the time evolution of locomotion parameters was similar to that of humans, which might contribute to understanding of adaptive mechanism in humans and to guiding principle for designing a control system of biped robots.

Typical methods for solving reinforcement learning problems iterate two steps, policy evaluation and policy improvement. This paper proposes algorithms for the policy evaluation to improve learning efficiency. The proposed algorithms are based on the Krylov Subspace Method (KSM), which is a nonstationary iterative method. The algorithms based on KSM are tens to hundreds times more efficient than existing algorithms based on the stationary iterative methods. Algorithms based on KSM are far more efficient than they have been generally expected. This paper clarifies what makes algorithms based on KSM makes more efficient with numerical examples and theoretical discussions.

In this paper, we present an experimental validation of a novel decentralized passivity-based control strategy for teleoperating a group of Unmanned Aerial Vehicles (UAVs): the slave side, consisting of the UAVs, is endowed with large group autonomy by allowing time-varying topology and interrobot/obstacle collision avoidance. The master side, represented by a human operator, controls the group motion and receives suitable force feedback cues informing her/him about the remote slave motion status. Passivity theory is exploited for guaranteeing stability of the slave side and of the overall teleoperation channel. Results of experiments involving the use of 4 quadcopters are reported and discussed, confirming the soundness of the paper theoretical claims.

In this paper, we investigate the locomotor behaviors of a biped robot on a splitbelt treadmill using a locomotion-control system composed of nonlinear oscillators with phase resetting. Our results show that the robot establishes stable walking on the treadmill at various speeds of the belts due to modulation of the rhythm and phase by phase resetting. In addition, the phase differences between the leg movements shifted from out of phase, and duty factors were autonomously modulated depending on the speed discrepancy between the belts occurring through dynamic interactions among the robot's mechanical system, the oscillator control system, and the environment. Such shifts of phase differences between the leg movements and modulations of duty factors are observed during human splitbelt treadmill walking, and our results suggest that our dynamic model using the robot and oscillator control system reflects a certain essence of the ability to produce adaptive locomotor behaviors.

In a general reinforcement learning problem, a plant (state transition probabilities) is estimated and a learning policy for the estimated plant is applied to a real plant. If there are differences between the estimated plant and the real plant, the obtained policy may not work for the real plant. Therefore, a set of plants with variations is used for learning in order to obtain a robust policy against variations. Bellman's principle of optimality does not hold when the set of plants is used, and a typical dynamic programming algorithm cannot solve the problem. This study shows the reason why the principle of optimality does not hold. It then makes some relaxed problems whose solutions can be obtained. Moreover, this study proposes solutions to learn feasible policies efficiently. The effectiveness of the proposed method is demonstrated by applying to simple examples.

This paper discusses a method to accelerate reinforcement learning. Firstly defined is a concept that reduces the state space conserving policy. An algorithm is then given that calculates the optimal cost-to-go and the optimal policy in the reduced space from those in the original space. Using the reduced state space, learning convergence is accelerated. Its usefulness for both DP (dynamic programming) iteration and Q-learning are compared through a maze example. The convergence of the optimal cost-to-go in the original state space needs approximately N or more times as long as that in the reduced state space, where N is a ratio of the state number of the original space to the reduced space. The acceleration effect for Q-learning is more remarkable than that for the DP iteration. The proposed technique is also applied to a robot manipulator working for a peg-in-hole task with geometric constraints. The state space reduction can be considered as a model of the change of observation, i.e., one of cognitive actions. The obtained results explain that the change of observation is reasonable in terms of learning efficiency.

This paper addresses an experimental system simulating a free-flying space robot, which has been constructed to study autonomous space robots. The experimental system consists of a space robot model, a frictionless table system, a computer system, and a vision sensor system. The robot model is composed of two manipulators and a satellite vehicle, and can move freely on a two-dimensional planar table, without friction, using air-bearings. The robot model has successfully performed the automatic truss structure assembly, including many jobs, e.g., manipulator berthing, component manipulation, arm trajectory control collision avoidance, assembly using force control, etc. Moreover, even if the robot fails in a task planned in advance, the robot re-plans the task by using reinforcement learning, and obtains the task goal for basically kinematic problems. But, for a class of complicated dynamic problems, the computational periods and efforts are infeasible for on-line learning. Some approaches are proposed to accelerate the learning speed, which also give models of cognitive actions and approaches to so-called a frame problem. The experiment demonstrates the possibility of the autonomous construction and the usefulness of space robots.

This paper addresses an experimental system simulating a free-flying space robot, which has been constructed to study autonomous space robots. The experimental system consists of a space robot model, a frictionless table system, a computer system, and a vision sensor system. The robot model composed of two manipulators and a satellite vehicle can move freely on a two-dimensional planar table without friction by using air-bearings. The robot model has successfully performed the automatic truss structure construction including many jobs, e.g., manipulator berthing, component manipulation, arm trajectory control avoiding collision, assembly considering contact with the environment, etc. Moreover, even if the robot fails in a task planned in advance, the robot accomplishes it by task re-planning through reinforcement learning. The experiment demonstrates the possibility of the automatic construction and the usefulness of space robots.

An experimental system simulating a free-flying space robot has been constructed to evaluate the performance and/or feasibility of control schemes. The experimental robot model, composed of a satellite base and a manipulator can move on a two-dimensional planar table without friction by using air-bearings. The effectiveness and control performance of various path control methods, i.e. local PD-control, resolved motion rate control and resolved acceleration control, are evaluated and compared with each other through numerical simulations and hardware experiments. The feasibility of the control schemes is examined. Autonomous capture using a moving and floating target is also demonstrated.

Two trajectory schemes far called rigid has been proposed. flexible are proposed by using a vir- In this paper, some trajectory schemes for the flex- tual rigid concept. A Re- ible are proposed by using the virtual rigid ma- solved Acceleration Control (RAC) fcr flexible ma- nipulator concept. A seudo Resolved Acceleration Control nipulators is presented. To suppress vibrating mo- (pseudo RAC) for flexible manipulators is presented. To tion, a composite of pseudo RAC and reduced- suppress vibrating motion a iicomposite of pseudo RAC and reduced-order modal control is developed. The order modai control is developed. The validity of the proposed schemes is explaiined through the singular perturbation method. The robust sta- bility of the pseudo RAC is proved. A sufficient condition for the asymptotic stability of the corn- posite is provided. The effectiveness of the proposed schemes is demonstrated through two case studies, i.e., one link flexible manipulator with a hardware demonstration and numerical ex- amples of a free-flying space robot with a flexible manipulator.

An experimental system is constructed to evaluate performance and feasibility of modeling, identification and control methods for space robots. The experimental system achieves a two-dimensional weightless condition by means of a horizontal table and air-pads. As a preliminary experimental study, two identification methods for space robots are discussed. The first identification method is a conventional method for manipulators on an inertially fixed base. The second identification method can determine dynamic parameters of space robots under the condition that the satellite base is free to both translate and rotate. This method is based upon the conservation laws of linear and angular momentum of a space robot. Finally implementation schemes of resolved acceleration control are considered for the experimental system.>