We propose VecNav, a novel approach that trains a monocular navigation model through self-supervision using uncalibrated, human-captured videos. These videos, characterized by unknown camera intrinsics and extrinsics, are readily available from video-sharing platforms (e.g. YouTube) and are referred to as "in-the-wild" videos due to their unregulated capture conditions. Our approach involves estimating ground truth trajectories from these videos using monocular visual odometry. We then train a transformer-based diffusion policy that takes a goal specified by a vector and RGB images as input and generates action predictions. Our method leverages a significantly larger and more diverse dataset compared to existing monocular visual navigation approaches. This diversity holds the potential to develop a generalist navigation model capable of guiding various types of robots in unfamiliar environments. We evaluated our method on a differential drive robot, demonstrating its capability to effectively navigate using solely "in-the-wild" videos for training. Our experiments demonstrate that VecNav successfully learned to act based on visual affordances, relying solely on uncalibrated "in-the-wild" data.

Recent advances in deep neural networks have opened up new possibilities for visuomotor robot learning. In the context of human-robot or robot-robot collaboration, such networks can be trained to predict future poses and this information can be used to improve the dynamics of cooperative tasks. This is important, both in terms of realizing various cooperative behaviors, and for ensuring safety. In this article, we propose a recurrent neural architecture, capable of transforming variable-length input motion videos into a set of parameters describing a robot trajectory, where predictions can be made after receiving only a few frames. A simulation environment is utilized to expand the training database and to improve generalization capability of the network. The resulting architecture demonstrates good accuracy when predicting handover trajectories, with models trained on synthetic and real data showing better performance than when trained on real or simulated data only. The computed trajectories enable the execution of handover tasks with uncalibrated robots, which was verified in an experiment with two real robots.

Model-based control has great potential for use in real robots due to its high sampling efficiency. Nevertheless, dealing with physical contacts and generating accurate motions are inevitable for practical robot control tasks, such as precise manipulation. For a real-time, model-based approach, the difficulty of contact-rich tasks that requires precise movement lies in the fact that a model needs to accurately predict forthcoming contact events within a limited length of time rather than detect them afterward with sensors. Therefore, in this study, we investigate whether and how neural network models can learn a task-related model useful enough for model-based control, that is, a model predicting future states, including contact events. To this end, we propose a structured neural network model predictive control (SNN-MPC) method, whose neural network architecture is designed with explicit inertia matrix representation. To train the proposed network, we develop a two-stage modeling procedure for contact-rich dynamics from a limited number of samples. As a contact-rich task, we take up a trackball manipulation task using a physical 3-DoF finger robot. The results showed that the SNN-MPC outperformed MPC with a conventional fully connected network model on the manipulation task.

Deep learning (DL) and reinforcement learning (RL) methods seem to be a part of indispensable factors to achieve human-level or super-human AI systems. On the other hand, both DL and RL have strong connections with our brain functions and with neuroscientific findings. In this review, we summarize talks and discussions in the "Deep Learning and Reinforcement Learning" session of the symposium, International Symposium on Artificial Intelligence and Brain Science. In this session, we discussed whether we can achieve comprehensive understanding of human intelligence based on the recent advances of deep learning and reinforcement learning algorithms. Speakers contributed to provide talks about their recent studies that can be key technologies to achieve human-level intelligence.

We propose a hierarchical control framework for generating versatile motions by a humanoid robot. The central feature of our framework is computational affordability: a large amount of computation time is allowable in the upper-level hierarchy. Consequently, whole-body trajectory optimization for a long time horizon becomes feasible. To ensure such affordability, a fast feedback loop is established in the lower-level hierarchy to increase the robustness against the large latency in the upper level. We experimentally examined the advantages of the achieved computational affordability. Our framework allowed a large computational time of 100 ms in each control cycle. This enables online trajectory optimization to predict 50 time steps ahead while taking full-body dynamics into account. Due to such a long prediction range, 20 motions were successfully generated in real time with our computation-ally affordable framework.

Our brain can be recognized as a network of largely hierarchically organized neural circuits that operate to control specific functions, but when acting in parallel, enable the performance of complex and simultaneous behaviors. Indeed, many of our daily actions require concurrent information processing in sensorimotor, associative, and limbic circuits that are dynamically and hierarchically modulated by sensory information and previous learning. This organization of information processing in biological organisms has served as a major inspiration for artificial intelligence and has helped to create in silico systems capable of matching or even outperforming humans in several specific tasks, including visual recognition and strategy-based games. However, the development of human-like robots that are able to move as quickly as humans and respond flexibly in various situations remains a major challenge and indicates an area where further use of parallel and hierarchical architectures may hold promise. In this article we review several important neural and behavioral mechanisms organizing hierarchical and predictive processing for the acquisition and realization of flexible behavioral control. Then, inspired by the organizational features of brain circuits, we introduce a multi-timescale parallel and hierarchical learning framework for the realization of versatile and agile movement in humanoid robots.

Previous works in the literature have claimed that the characteristics of electromyography (EMG) signals depend on each person, and thus, EMG interfaces need to be carefully calibrated for each user in myoelectric control. In this study, we show that the EMG interface used to estimate the joint torques of a user can be constructed simply by incorporating other users’ data without typical calibration process. To achieve this plug-and-play capability, we introduce the concept of collaborative filtering to estimate the joint torque of a novel user by exploiting the preidentified relationships between motion-body features, including EMG signals, and the joint torques of other users. To validate our proposed approach, we compare the performance of estimating joint torque by the proposed method with that by conventional linear regression models as a baseline. We considered the following two baseline methods.Linear-own:The parameters of the linear model are calibrated for each subject from his/her own training data.Linear-others:The parameters of the linear model are calibrated with the other users’ data in which the novel user's data are not included. As a result, the estimated joint torques from our proposed approach reveal a better estimation performance than those from the baseline approaches. Furthermore, we also successfully demonstrate online myoelectric control of an upper limb exoskeleton robot with an attached mannequin arm.

Dynamic movement primitives (DMPs) have proven to be an effective movement representation for motor skill learning. In this paper, we propose a new approach for training deep neural networks to synthesize dynamic movement primitives. The distinguishing property of our approach is that it can utilize a novel loss function that measures the physical distance between movement trajectories as opposed to measuring the distance between the parameters of DMPs that have no physical meaning. This was made possible by deriving differential equations that can be applied to compute the gradients of the proposed loss function, thus enabling an effective application of backpropagation to optimize the parameters of the underlying deep neural network. While the developed approach is applicable to any neural network architecture, it was evaluated on two different architectures based on encoder-decoder networks and convolutional neural networks. Our results show that the minimization of the proposed loss function leads to better results than when more conventional loss functions are used.

In this study, we propose a novel human-in-the-loop optimization approach for exoskeleton robot control. We develop a method to optimize widely-used Electromyography (EMG)-based assistive strategies. If we use multiple EMG channels to control multi-DoF robots, optimization process becomes complex and requires a large amount of data. To make the optimization tractable, we exploit the synergies both of the human muscles and artificial muscles of the exoskeleton robots to reduce the number of parameters of the assistive strategies. We show that we can extract the synergies not only from the user's muscle activities but from pneumatic artificial muscle (PAMs) contractions of the exoskeleton robot. Then, we adopt a Bayesian optimization method to acquire the parameters for assisting human movements by iteratively identifying the user's preferences of the assistive strategies. We conducted experiments to evaluate our proposed method with a PAMs-driven upper-limb exoskeleton robot. Our method successfully learned assistive strategies from the human-in-theloop optimization with a practicable number of interactions.

Learning to recognize and reproduce handwritten characters is already a challenging task both for humans and robots alike, but learning to do the same thing for characters that can be transformed arbitrarily in space, as humans do when writing on a blackboard for instance, significantly ups the ante from a robot vision and control perspective. In previous work we proposed various different forms of encoder-decoder networks that were capable of mapping raw images of digits to dynamic movement primitives (DMPs) such that a robot could learn to translate the digit images into motion trajectories in order to reproduce them in written form. However, even with the addition of convolutional layers in the image encoder, the extent to which these networks are spatially invariant or equivariant is rather limited. In this paper, we propose a new architecture that incorporates both an image-to-motion encoder-decoder and a spatial transformer in a fully differentiable overall network that learns to rectify affine transformed digits in input images into canonical forms, before converting them into DMPs with accompanying motion trajectories that are finally transformed back to match up with the original digit drawings such that a robot can write them in their original forms. We present experiments with various challenging datasets that demonstrate the superiority of the new architecture compared to our previous work and demonstrate its use with a humanoid robot in a real writing task.

This paper proposes a novel controller framework for antagonistically driven pneumatic artificial muscle (PAM) actuators. The proposed controller can be stably configured in both torque-stiffness control and position-stiffness control modes. Three contributions are sequentially presented in constructing the framework: 1) A PAM force feedback controller with guaranteed stability is synthesized in a way so as to contend with nonlinear PAM characteristics; 2) a mathematical tool is developed to compute reference PAM forces, for a given set of desired joint torque and joint stiffness inputs; and 3) on top of the torque controller, a position control scheme is implemented and its stability analysis is given in the sense of Lyapunov. In order to test the controller framework, an extensive set of experiments are conducted using an actuator that is constructed using two antagonistically coupled PAMs. As a result, the actuator exhibits satisfactory tracking performances concerning both torque-stiffness control and position-stiffness control modes.

This paper introduces an approach that robotizes an ankle-foot orthosis (AFO). In particular, toward post-stroke gait rehabilitation, we robotize a double-bar AFO, which is widely used in rehabilitation facilities, by newly designing a modular joint, a pneumatic actuator, and a Bowden cable force-transmission system. Our modular joint system, called the Modular Exoskeletal Joint (MEJ), has a hollow shaft for simple attachment to an AFO's pivot. We designed MEJ to compactly house an encoder that is built in a bearing in a pulley. We adopted Bowden cables to transmit contraction forces from an actuator to the MEJ. As an actuation scheme, we developed the Nested-cylinder Pneumatic Artificial Muscle (NcPAM) system. Even though PAMs are mechanically compliant and lightweight, they can still generate a large force. Therefore, they can provide an ideal actuation system for exoskeletal robots. The nested-cylinder in NcPAM houses a cable-tensioning spring to properly maintain small cable tension for passive movements and a cable stopper to connect the PAM and the cable for properly transmitting the large force generated by PAM. We show the ankle-joint trajectory tracking performances of this integrated system using iterative learning control.

In this paper we propose a new approach for learning perception-action couplings. We show that by collecting a suitable set of raw images and the associated movement trajectories, a deep encoder-decoder network can be trained that takes raw images as input and outputs the corresponding dynamic movement primitives. We propose suitable cost functions for training the network and describe how to calculate their gradients to enable effective training by back-propagation. We tested the proposed approach both on a synthetic dataset and on a widely used MNIST database to generate handwriting movements from raw images of digits. The calculated movements were also applied for digit writing with a real robot.

Since exoskeletons show potential for rehabilitation therapy, many scientists have been designing upper extremity exoskeletons. Unfortunately, few have successfully provided a shoulder exoskeleton for severe impairment. Toward Brain-Machine-Interface (BMI) rehabilitation robot therapies for severe upper extremity impairment, this paper introduces a shoulder exoskeleton robot with a modular joint and an off-board modular actuator. We applied a Modular Exoskeletal Joint (MEJ) to a shoulder exoskeleton that was driven by Pneumatic Artificial Muscles (PAMs) transmitted by a Bowden cable. Our objective is generating passive movements triggered by BMI. Since large torque has to be generated for assist a whole arm, we newly designed a more powerful Nested-cylinder PAMs (NcPAMs) than our previous work. As a proof of the concept of the mechatronics design, we show the tracking performance of the periodic trajectory of a joint angle with both human and mannequin arms as simulated impairments. Our result shows that the tracking error is sufficiently small in all of the conditions and that our developed shoulder exoskeleton is an adequate substitute for flexion/extension movements.

Haptic guidance is a motor training procedure in which the subject is physically guided through an ideal motion by an haptic interface. It is especially valuable for repetitive training, and more effective when coupled with additional sensory feedback, e.g. the visual modality. The advantage of additional feedback modalities may stem from the learner's increased active participation and engagement. Here, we test this hypothesis by analyzing the learners' brain state during haptic training. Specifically, we focus on the sensorimotor rhythms (SMR), since they have been previously associated with the level of engagement directed towards a motor task. We conducted a circle-drawing haptic training, where subjects were asked to memorise the guided trajectory while their electroencephalogram (EEG) was recorded. During the experiment, only the haptic modality was maintained (i.e. unimodal) by keeping the task workspace visually hidden. Results show a clear trend: subjects who exhibited a performance improvement, were characterized by a stronger desynchronization of the beta rhythms over the contralateral hemisphere. This is in agreement with recent studies showing that contralateral beta rhythms changes are associated with motor skills retention. Moreover, under the assumption that SMR are indeed a marker of engagement, our results represent accumulating evidence that active participation is crucial for haptic training.

Humans use multiple muscles to generate such joint movements as an elbow motion. With multiple lightweight and compliant actuators, joint movements can also be efficiently generated. Similarly, robots can use multiple actuators to efficiently generate a one degree of freedom movement. For this movement, the desired joint torque must be properly distributed to each actuator. One approach to cope with this torque distribution problem is an optimal control method. However, solving the optimal control problem at each control time step has not been deemed a practical approach due to its large computational burden. In this paper, we propose a computationally efficient method to derive an optimal control strategy for a hybrid actuation system composed of multiple actuators, where each actuator has different dynamical properties. We investigated a singularly perturbed system of the hybrid actuator model that subdivided the original large-scale control problem into smaller subproblems so that the optimal control outputs for each actuator can be derived at each control time step and applied our proposed method to our pneumatic-electric hybrid actuator system. Our method derived a torque distribution strategy for the hybrid actuator by dealing with the difficulty of solving real-time optimal control problems.

The paper presents a novel control method for the arm exoskeletons that takes into account the muscular force manipulability of the human arm. In contrast to classical controllers that provide assistance without considering the biomechanical properties of the human arm, we propose a control method that takes into account the configuration of the arm and the direction of the motion to effectively compensate the anisotropic property of the muscular manipulability of the human arm. Consequently, the proposed control method effectively maintains a spherical endpoint manipulability in the entire workspace of the arm. As a result, the proposed method allows the human using the exoskeleton to efficiently perform tasks in arm configurations that are normally unsuitable due to the low manipulability. We evaluated the proposed approach by a preliminary experimental study where a subject wearing a 2 DOF arm-exoskeleton had to move a 4 kg weight between several locations. The results of our study demonstrate that the proposed approach effectively augments the ability of human motor control to perform tasks equally well in the whole arm workspace that include configurations with low intrinsic manipulability.

Recent studies suggest that reinforcement learning has great potential for generating assistive strategies in exoskeletons through physical interactions between a user and a robot. Previous methods focused on a task-specific assistive strategy, where for every single task (situation/context), the user needs to interact with a robot to learn an appropriate assistive strategy. Therefore, the learned strategies cannot be generalized for a new task. Since the sampling cost is expensive for such human-in-the-loop systems as exoskeletons, generalization must be enabled. In this paper, we propose to learn task-parametrized assistive strategies for exoskeleton robots. Our method employs an assistive strategy, which depends on the task parameter and the state variable, that can be learned from multiple sets of human-robot interaction data across different tasks and generalized even for an unseen task, given the task parameter without additional learning. To alleviate the user's burden in the learning process across multiple tasks, we exploit a data-efficient multi-task reinforcement learning framework. To verify the effectiveness of our method, we developed an experimental platform with an exoskeleton robot. We conducted a series of experiments whose experimental results show that our method can learn such a task-parametrized assistive strategy and be generalized for unseen tasks to reduce the user's electromyography signals (EMGs) during tasks.

Pneumatic Artificial Muscle (PAM) actuators have been used as exoskeletons because of their inherited compliance and high power-weight ratio. However, creating accurate models remains difficult mainly due to the compliance issue; the model can be changed by the force applied by the user. Therefore, both user and robot actions need to be considered for sufficient excitation of PAMs that are equipped in exoskeleton robots, unlike typical rigid actuators that can only be sufficiently excited by robot actions. In this paper, we propose a user-robot collaborative excitation approach for PAM model identification as an active learning framework for sequentially collecting data by deriving and executing optimal user and robot actions at each step with Gaussian processes. The optimal actions, which are executed by the robot, are displayed on a monitor that enables the user to execute them. We conducted experiments using a powered elbow exoskeleton with a PAM actuator. Experimental results show that our method can more efficiently identify the PAM model than a standard model identification method that does not use any data acquired through user-robot collaboration.

Social demand for exoskeleton robots that physically assist humans has been increasing in various situations due to the demographic trends of aging populations. With exoskeleton robots, an assistive strategy is a key ingredient. Since interactions between users and exoskeleton robots are bidirectional, the assistive strategy design problem is complex and challenging. In this paper, we explore a data-driven learning approach for designing assistive strategies for exoskeletons from user-robot physical interaction. We formulate the learning problem of assistive strategies as a policy search problem and exploit a data-efficient model-based reinforcement learning framework. Instead of explicitly providing the desired trajectories in the cost function, our cost function only considers the user’s muscular effort measured by electromyography signals (EMGs) to learn the assistive strategies. The key underlying assumption is that the user is instructed to perform the task by his/her own intended movements. Since the EMGs are observed when the intended movements are achieved by the user’s own muscle efforts rather than the robot’s assistance, EMGs can be interpreted as the “cost” of the current assistance. We applied our method to a 1-DoF exoskeleton robot and conducted a series of experiments with human subjects. Our experimental results demonstrated that our method learned proper assistive strategies that explicitly considered the bidirectional interactions between a user and a robot with only 60 seconds of interaction. We also showed that our proposed method can cope with changes in both the robot dynamics and movement trajectories.

In this study, we propose a human movement model both for myoelectric assistive robot control and biosignal-sensor-failure detection. We particularly consider an application to upper extremity exoskeleton robot control. When using electromyography (EMG)-based assistive robot control, EMG electrodes can be easily disconnected or detached from skin surfaces because the human body is always in contact with the robot. If multiple electrodes are used to estimate multiple joint movements, the probability of sensor electrode misplacement increases due to human error. To cope with the aforementioned issues, we propose a novel human movement estimation model that takes anomalies into account as uncertain observations. We estimated human joint torques by automatically modulating the contribution of each sensor channel for the movement estimation based on anomaly scores that were computed according to synergistic muscular coordination. We compared our proposed method with conventional approaches during drinking-movement estimation with five healthy subjects in the three aforementioned anomaly situations and showed the effectiveness of our proposed method. We applied it to a four-DOF upper limb assistive exoskeleton robot and showed proper control in sensor failure situations.

Designing an assistive strategy for exoskeletons is a key ingredient in movement assistance and rehabilitation. While several approaches have been explored, most studies are based on mechanical models of the human user, i.e., rigid-body dynamics or Center of Mass (CoM)-Zero Moment Point (ZMP) inverted pendulum moECenter of Massdel, or only focus on periodic movements with using oscillator models. On the other hand, the interactions between the user and the robot are often not considered explicitly because of its difficulty in modeling. In this paper, we propose to learn the assistive strategies directly from interactions between the user and the robot. We formulate the learning problem of assistive strategies as a policy search problem. To alleviate heavy burdens to the user for data acquisition, we exploit a data-efficient model-based reinforcement learning framework. To validate the effectiveness of our approach, an experimental platform composed of a real subject, an electromyography (EMG)-measurement system, and a simulated robot arm is developed. Then, a learning experiment with the assistive control task of the robot arm is conducted. As a result, proper assistive strategies that can achieve the robot control task and reduce EMG signals of the user are acquired only by 30 seconds interactions.

This paper introduces a novel asynchronous adaptive brain machine interface (BMI), based on a dry-wireless headset, to trigger the movement of a lower limb exoskeleton robot by foot motor imagery. Specifically, it addresses two issues that are critical for the development of a plug-and-play brain robot interface (BRI): setup-time and the nonstationarity of the electroencephalogram (EEG). The former is solved by a dry-wireless headset that reduces setup-time compared to gel-based systems, and removes the nuisance of cables. The latter has been extensively studied in the literature, leading to effective adaptive algorithms in synchronous BMI. However, asynchronous BMI has received little attention. We propose an extension of state-of-the-art adaptive methods by defining the forgetting factors according to the time constant of the exponential moving average. In addition, we propose feature adaptation as opposed to the standard bias adaptation of a linear classifier. After calibrating the decoder, the subject with a reliable classification of sensorimotor rhythms was asked to trigger robot squatting. The motion was successfully initialized by foot motor imagery; with an essential contribution of the proposed adaptive BMI, which makes features less prone to nonstationarities and improves classification performance compared to standard adaptive methods. The ultimate goal of our research is to develop a plug-and-play co-adaptive BRI for neuromotor rehabilitation.

An effective robot trajectory representation should encode all relevant aspects of the desired motion. For kinematic representations, this means that both the spatial course of the trajectory and its speed profile must be specified. The concept of dynamic movement primitives (DMP) provides a kinematic representation that fully specifies these two aspects of motion. They are, however, not separated from each other within the DMP representation. This can be problematic when movements with significant speed variations are compared within movement recognition and skill learning algorithms. In such comparisons it is often important to distinguish between the spatial and temporal aspects of motion. In this paper we propose a new representation based on dynamic movement primitives, where spatial and temporal aspects are well separated. We demonstrate the effectiveness of the proposed representation for statistical learning of robot skills and movement recognition and compare the performance with standard DMPs, where temporal and spatial aspects of motion are intertwined.

This paper presents a biologically-inspired real-time balance recovery control strategy that is applied to a lower body exoskeleton with variable physical stiffness actuators at its ankle joints. For this purpose, a torsional spring-loaded flywheel model is presented to encapsulate both approximated angular momentum and variable physical stiffness, which are crucial parameters in describing the postural balance. In particular, the incorporation of physical compliance enables us to provide three main contributions: i) A mathematical formulation is developed to express the relation between the dynamic balance criterion ZMP and the physical ankle joint stiffness. Therefore, balancing control can be interpreted in terms of ankle joint stiffness regulation. ii) ‘Variable physical’ stiffness is utilized in the bipedal robot balance control task for the first time in the literature, to the authors' knowledge. iii) The variable physical stiffness strategy is compared with the optimal constant stiffness strategy by conducting experiments on our exoskeleton robot. The results indicate that the proposed method provides a favorable balancing control performance to cope with unperceived perturbations, in terms of center of mass position regulation, ZMP error and mechanical power.

In this paper, we propose an estimation method of human joint movements from measured EMG signals for assistive robot control. We focus on how to estimate joint movements using multiple EMG electrodes even under sensor failure situations. In real world applications, EMG sensor electrodes might become disconnected or detached from skin surfaces. If we consider EMG-based robot control for assistive robots, such sensor failures lead to significant errors in the estimation of user joint movements. To cope with these sensor failures, we propose a state estimation model that takes uncertain observations into account. Sensor channel anomalies are found by checking the covariance of the EMG signals measured by multiple EMG electrodes. To validate the proposed control framework, we artificially disconnect an EMG electrode or detach one side of an EMG probe from the skin surface during elbow joint movement estimation. We show proper control of a one-DOF exoskeleton robot based on the estimated joint torque using our proposed method even when one EMG electrode has a sensor problem; a standard method with no tolerability against uncertain observations was unable to deal with these fault situations. Furthermore, the errors of the estimated joint torque with our proposed method were smaller than the standard method or a method with a conventional sensor fault detection algorithm.

Human-demonstrated motion transferred to a robotic platform often needs to be adapted to the current state of the environment or to modified task requirements. Adaptation, i. e. learning of a modified behavior, needs to be fast to enable quick utilization of the robot either in industry or in future household-assistant tasks. In this paper we show how to accelerate trajectory adaptation based on learning of coupling terms in the framework of dynamic movement primitives (DMPs). Our method applies ideas from feedback error learning to iterative learning control (ILC). By taking into account the actual physical constraints of the synchronous motion - through synchronization of both positions (or forces) and velocities - it is not only a more faithful representation of actual real-world processes, but it also accelerates the speed of convergence. To show the applicability of the approach in the framework of DMPs, we tested it on a formulation which encodes an initial discrete motion, followed by a periodic behavior, all in a single system. Modifications of the original discrete-periodic formulation now also allow for the use of DMP temporal scaling property. In the paper we also show how the DMP coupling can be implemented in joint space, whereas the measured forces and previous approaches always remained in the task space. We applied our approach to an example dual-arm synchronization task on Sarcos humanoid robot CB-i.

Conventional hybrid actuators can achieve a better force/torque bandwidth than a single principle actuator without losing back-drivability. However, hybrid actuators occupy space at least equal to the sum of the volumes of two or more actuators and multiple transmissions consisting of timing belts, gears, links, tendons, or Bowden-cables. Designs in which space and structural elements of a hybrid actuator are integrated have not been sufficiently studied. This paper presents an original development work on the design of an integrated pneumatic-electromagnetic hybrid linear actuator (iPEHLA). In the integrated design, hybrid components of an air cylinder and a linear motor were arranged around a single shaft, and the pneumatic and electromagnetic actuators shared the same moving parts (i.e., piston and moving part) and internal cylindrical spaces (i.e., cylinder and stator). Consequently, space required was less than the sum of the volumes of the two actuators. To verify the feasibility of this novel integrated design, the iPEHLA was designed and prototyped by modifying parts of a commercially available pneumatic cylinder integrated with electromagnetic components. The performances of the iPEHLA under pneumatic actuation and hybrid actuation were compared with respect to a feedforward friction compensation. The experimental results demonstrated that the novel design maintained its hybrid properties.

This paper describes a novel controller that is capable of simultaneously controlling torque and variable stiffness in real-time, for actuators with antagonistically driven pneumatic artificial muscles (PAMs). To this end, two contributions are presented: i) A stable force feedback controller that can cope with inherent PAM nonlinearities is synthesized using the dissipativity theory, for each PAM unit. ii) On top of this force feedback controller, a mathematical formulation is developed to compute reference force inputs that correspond to desired joint torque and joint stiffness inputs, concerning both agonist and antagonist PAMs. This strategy enables us to introduce real-time sensory feedback; torque and stiffness control is addressed by means of PAM force feedback control with guaranteed stability. To validate the proposed control scheme, a series of experiments were conducted on an experimental setup. As the result, the controller exhibited favorable torque and stiffness tracking in real-time, demonstrating that it could meet the performance criteria to power exoskeleton systems.

The creation and adaptation of motor behaviors is an important capability for autonomous robots. In this paper we propose an approach for altering existing robot behaviors online, where a human coach interactively changes the robot motion to achieve the desired outcome. Using hand gestures, the human coach can specify the desired modifications to the previously acquired behavior. To preserve a natural posture while performing the task, the movement is encoded in the robot's joint space using periodic dynamic movement primitives. The coaching gestures are mapped to the robot joint space via robot Jacobian and used to create a virtual force field affecting the movement. A recursive least squares technique is used to modify the existing movement with respect to the virtual force field. The proposed approach was evaluated on a simulated three degrees of freedom planar robot and on a real humanoid robot, where human coaching gestures were captured by an RGB-D sensor. Although our focus was on rhythmic movements, the developed approach is also applicable to discrete (point-to-point) movements.

In this paper, we propose an optimal control framework for pneumatic actuators. In particular, we consider using Pneumatic Artificial Muscle (PAM) as a part of Pneumatic-Electric (PE) hybrid actuation system. An optimal control framework can be useful for PE hybrid system to properly distribute desired torque outputs to the actuators that have different characteristics. In the optimal control framework, the standard choice to represent control cost is squared force or torque outputs. However, since the control input for PAM is pressure rather than the force or the torque, we should explicitly consider the pressure of PAM as the control cost in an objective function of the optimal control method. We show that we are able to use pressure input as the control cost for PAM by explicitly considering the model which represents a relationship between the pressure input and the force output of PAM. We demonstrate that one-DOF robot with the PE hybrid actuation system can generate pressure-optimized ball throwing movements by using the optimal control method.

Dynamic movement primitives (DMPs) were proposed as an efficient way for learning and control of complex robot behaviors. They can be used to represent point-to-point and periodic movements and can be applied in Cartesian or in joint space. One problem that arises when DMPs are used to define control policies in Cartesian space is that there exists no minimal, singularity-free representation of orientation. In this paper we show how dynamic movement primitives can be defined for non minimal, singularity free representations of orientation, such as rotation matrices and quaternions. All of the advantages of DMPs, including ease of learning, the ability to include coupling terms, and scale and temporal invariance, can be adopted in our formulation. We have also proposed a new phase stopping mechanism to ensure full movement reproduction in case of perturbations.

In this paper, we introduce our ongoing work on the development of an upper body exoskeleton robot, driven by a pneumatic-electric hybrid actuation system. Since the limb of an exoskeleton robot needs to have small inertia to achieve agility and safety, using a heavy actuator is not preferable. Furthermore, we need to use backdrivable actuators that can generate sufficiently large torques to support user movements. These two requirements may seem contradictory. In order to cope with this development problem, we use a hybrid actuation system composed of Pneumatic Artificial Muscles (PAMs) and small-size electromagnetic motors. Although we and other research groups have already presented the advantage of the hybrid actuation system, we newly propose the usage of Bowden cable in a hybrid actuator to transmit the force generated by the PAMs to joints of our exoskeleton robot so that we can design a compact upper limb with small inertia. In addition, small size electric motors are mechanically connected to joints in order to compensate uncertainty generated by the PAM dynamics and the Bowden cable. We demonstrate that the proposed joint is backdrivable with the capability of large torque generation for the gravity compensation task both in One-DOF system with a dummy weight and right arm of the upper body exoskeleton with a mannequin arm. We also show the right arm exoskeleton can be moved using a torque input, extracted from sensory information via a goniometer.

This study proposes the design of electromyography (EMG)-based force feedback controller which explicitly considers human-robot interaction for the exoskeletal assistive robot. Conventional approaches have been only consider one-directional mapping from EMG to control input for assistive robot control. However, EMG and force generated by the assistive robot interfere each other, e.g., amplitude of EMG decreases if limb movements are assisted by the robot. In our proposed method, we first derive the nonlinear mapping from EMG signal to muscle force for estimating human joint torque, and convert it to assistive force using human musculoskeletal model and robot kinematic model. Additionally the feedforward interaction torque is feedback into torque controller to acquire the necessity loads. To validate the feasibility of the proposed method, assistive One-DOF system was developed as the real equipment and the simulator. We compared the proposed method with conventional approaches using both the simulated and the real One-DOF systems. As the result, we found that the proposed model was able to estimate the necessary torque adequately to achieve stable human-robot interaction.

Synergy is a key concept for understanding smooth and dexterous body movements generated in biological systems. In this paper, we propose an approach based on canonical correlation analysis (CCA) for identifying synergies that capture coherences organized between motor activations and sensory signals in movements. Using a musculoskeletal planar arm model with muscle and joint redundancies, we show that synergies identified by CCA give more natural and tractable activation patterns in virtual controller space with lower dimensionality when the arm performs reaching and a 3D pole balancing movement tasks.

We develop fast reinforcement learning (RL) framework using the approximated dynamics of a humanoid robot. Although RL is a useful non-linear optimizer, applying it to real robotic systems is usually difficult due to the large number of iterations required to acquire suitable policies. In this study, we approximate the dynamics using data from a real robot with sparse pseudo-input Gaussian processes (SPGPs). By using SPGPs, we estimated the probability distribution considering both the input vector and output signal variances. In real environments, since the observations from robotic sensors include large noise, SPGPs can suitably approximate the stochastic dynamics of a real humanoid robot. We use the approximated dynamics to improve the performance of a movement task in a path integral RL framework, which updates a policy from the sampled trajectories of the state and action vectors and the cost. We implemented our proposed method on a real humanoid robot and tested on a via-point reaching task. The robot achieved successful performance with fewer number of interactions with the real environment by using the proposed method than a conventional approach which dose not use the simulated dynamics.

Direct transfer of human motion trajectories to humanoid robots does not result in dynamically stable robot movements due to the differences in human and humanoid robot kinematics and dynamics. We developed a system that converts human movements captured by a low-cost RGB-D camera into dynamically stable humanoid movements. The transfer of human movements occurs in real-time. As need arises, the developed system can smoothly transition between unconstrained movement imitation and imitation with balance control, where movement reproduction occurs in the null space of the balance controller. The developed balance controller is based on an approximate model of the robot dynamics, which is sufficient to stabilize the robot during on-line imitation. However, the resulting movements cannot be guaranteed to be optimal because the model of the robot dynamics is not exact. The initially acquired movement is therefore subsequently improved by model-free reinforcement learning, both with respect to the accuracy of reproduction and balance control. We present experimental results in simulation and on a real humanoid robot.

The paper reports on a novel hybrid drive lower-extremity exoskeleton research platform, XoR2, an improved version of XoR. Its design concept, details of the new hardware and basic experimental results are presented. The robot is designed so that it does not interfere with the user's normal walking and supports a 30-kg payload in addition to its own weight of 20 kg. The robot has a total of 14 joints; among them six flexion/extension joints are powered. Pneumatic artificial muscles are combined with small high-response servo motors for the hip and knee joints, and arranged antagonistically at the hip and ankle joints to provide passive stability and variable stiffness. The preliminary experimental results on position and torque control demonstrate that the proposed mechanisms, sensors and control systems are effective, and hybrid drive is promising for torque-controllable, high-speed, backdrivable, mobile (but non-power-autonomous) exoskeleton robots.

We introduce our Pneumatic-Electric (PE) hybrid actuator model and propose to use the model to derive a controller for the hybrid actuation system by an optimal control method. Our PE hybrid actuator is composed of Pneumatic Artificial Muscle (PAM) and an electric motor. The PE hybrid actuator is light and can generate large torque. These properties are desirable for assistive devices such as exoskeleton robots. However, to maximally take advantage of PE hybrid system, we need to reasonably distribute necessary torque to these redundant actuators by properly taking distinctive characteristics of a pneumatic actuator and an electric motor into account. To do this, in this study, we use an optimal control method called iterative LQG to reasonably distribute the necessary torque to the PAM and the electric motor. The crucial issue to apply the optimal control method to the PE hybrid system is PAM modeling. We built a PAM model composed of three elements: 1) an (air)pressure-force conversion model, 2) a contraction rate model, 3) time delay of the air valve, and 4) the upper limit of force generation that depends on the contraction rate and the movable range. We apply our proposed method to a one degree of freedom (one-DoF) arm with PE hybrid actuator. The one-DoF arm successfully swing tasks 0.5 Hz, 2 Hz and 4 Hz and swing up and stability task by reasonably distributing necessary torque to the two different actuators in a simulated and a real environments.

Learning of goal-directed behaviors in biological systems is broadly based on associations between conditional and unconditional stimuli. This can be further classified as classical conditioning (correlation-based learning) and operant conditioning (reward-based learning). Although traditionally modeled as separate learning systems in artificial agents, numerous animal experiments point towards their co-operative role in behavioral learning. Based on this concept, the recently introduced framework of neural combinatorial learning combines the two systems where both the systems run in parallel to guide the overall learned behavior. Such a combinatorial learning demonstrates a faster and efficient learner. In this work, we further improve the framework by applying a reservoir computing network (RC) as an adaptive critic unit and reward modulated Hebbian plasticity. Using a mobile robot system for goal-directed behavior learning, we clearly demonstrate that the reservoir critic outperforms traditional radial basis function (RBF) critics in terms of stability of convergence and learning time. Furthermore the temporal memory in RC allows the system to learn partially observable markov decision process scenario, in contrast to a memory less RBF critic.

The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge is how to reduce the variance of policy gradient estimates for reliable policy updates. In this letter, we combine the following three ideas and give a highly effective policy gradient method: (1) policy gradients with parameter-based exploration, a recently proposed policy search method with low variance of gradient estimates; (2) an importance sampling technique, which allows us to reuse previously gathered data in a consistent way; and (3) an optimal baseline, which minimizes the variance of gradient estimates with their unbiasedness being maintained. For the proposed method, we give a theoretical analysis of the variance of gradient estimates and show its usefulness through extensive experiments.

In this paper, we propose a framework for generating coordinated periodic movements of robotic systems with external inputs. We developed an adaptive pattern generator model that is composed of a two-factor observation model with a style parameter and phase dynamics with a phase variable. The style parameter controls the spatial patterns of the generated trajectories, and the phase variable controls its temporal profiles. To validate the effectiveness of our proposed method, we applied it to a simulated humanoid model to perform biped walking behaviors coordinated with observed walking patterns and the environment. The robot successfully performed stable biped walking behaviors even when the style of the observed walking pattern and the period were suddenly changed.

In this study, we propose a control method for movement assistive robots using measured signals from human users. Some of the wearable assistive robots have mechanisms that can be adjusted to human kinematics (e.g., adjustable link length). However, since the human body has a complicated joint structure, it is generally difficult to design an assistive robot which mechanically well fits human users. We focus on the development of a control algorithm to generate corresponding movements of wearable assistive robots to that of human users even when the kinematic structures of the assistive robot and the human user are different. We first extract the latent kinematic relationship between a human user and the assistive robot. The extracted relationship is then used to control the assistive robot by converting human behavior into the corresponding joint angle trajectories of the robot. The proposed approach is evaluated by a simulated robot model and our newly developed exoskeleton robot.

Disturbance rejection is one of the most important abilities required for biped walkers. In this study, we propose a method for dynamic programming of biped walking and apply it to a simple passive dynamic walker (PDW) on an irregular slope. The key of the proposed approach is to employ the transient dynamics of the walker just before approaching the falling state in the absence of any controlling input, and to derive the optimal control policy in the low-dimensional latent space. In recent our study, we found that such transient dynamics deeply relates to the basin of attraction for a stable gait. By patching latent coordinates to such a structures in each Poincaré section and defining the reward function according to the survive time of the transient dynamics, so-called escape-times, we construct a Markov decision process (MDP) for the PDW and obtain an optimal policy using a dynamic programming (DP). We will show that the proposed method actually succeeds in controlling the PDW even if the degree of disturbance is relatively large and the dimensionality of coordinates is reduced to lower ones.

Learning about new objects that a robot sees for the first time is a difficult problem because it is not clear how to define the concept of object in general terms. In this paper we consider as objects those physical entities that are comprised of features which move consistently when the robot acts upon them. Among the possible actions that a robot could apply to a hypothetical object, pushing seems to be the most suitable one due to its relative simplicity and general applicability. We propose a methodology to generate and apply pushing actions to hypothetical objects. A probing push causes visual features to move, which enables the robot to either confirm or reject the initial hypothesis about existence of the object. Furthermore, the robot can discriminate the object from the background and accumulate visual features that are useful for training of state of the art statistical classifiers such as bag of features.

We propose an adaptive walking assistance strategy to control an exoskeleton robot. In our proposed framework, we explicitly consider the following: 1) the diversity of user motions (style) and 2) the interactions among a user, a robot, and an environment. To spatially coordinate a wide variety of user motions and robot behaviors, we estimated style parameters from observed user movements. To temporally coordinate the interactions among the user, the robot, and the environment, we synchronized the phases of these three systems with a coupled oscillator model. The estimated style parameters and the phase of the user motion can be used to predict future user movements. We investigated how movement prediction and phase synchronization can be beneficial to control an exoskeleton robot. To evaluate our adaptive walking assistance strategy, we developed simulated user and exoskeleton models. The physical interactions among the user, the exoskeleton, and the ground models are introduced in the simulated system. We show that the necessary torque for the user walking movement was reduced around 40% by using our proposed method to control the exoskeleton model.

We propose a novel exoskeleton robot prototype aimed at a brain-machine interface and rehabilitation for postural control for elderly people, people with spinal cord injury, stroke patients, and others with similar needs. By arranging pneumatic muscles with electric motors in a optimal way, one can achieve both weight-reduction and torque-controllability. Its anthropomorphic design and torque-controllability enable users to implement and test various rehabilitation/compensation programs consistent with human motor control and learning mechanism. Hybrid drive itself is not new, but its specialized application to lightweight exoskeleton is novel. This paper reports the design and development of the robot, particularly addressing a hybrid drive for load-bearing tasks such as standing and postural maintenance. The experimental data as well as the attached videos demonstrate the effectiveness of the proposed system.

Falls that occur during walking are a significant problem from the viewpoints of both medicine and robotics engineering. It is very important to predict falls in order to prevent the falls or minimize the ensuing damage from them. In this study, we investigate the structure of the escape-times from walking to falling of a passive dynamic biped walker on a slope in a 2D plane with irregularities. We find that the structure lies on a manifold with high nonlinearity in state space that cannot be analyzed by linear methods under the assumption of a Gaussian distribution. Therefore, we first apply an extension of the support vector machine (SVM) to characterize its nonlinear structure, which enables us to predict imminent falls. Next, we find a latent space which describes the essential dynamics of the passive walker in a lower-dimensional space using canonical correlation analysis (CCA). There is wide applicability of this work for monitoring walking anomalies of both robots and human beings.

In this study, we propose novel modular architecture to control a robot with many degrees of freedom, such as a humanoid robot. High-degree-of-freedom (DOF) robots tend to have highly nonlinear dynamics. In general, deriving a nonlinear controller for high-dimensional systems is intractable. In our approach, we adopt multiple Linear Quadratic Gaussian (LQG) controllers to cope with nonlinear dynamics. Switching criteria for the modular architecture is provided by Bellman's optimality. The proposed method is applied to a simulated 10-DOF biped model and a 51-DOF humanoid robot called CB-i.

We present our ongoing effort to achieve compliant balancing to dynamic walking on our torque-controlled, human-sized, biped humanoid robot. Inspired by human musculoskeletal systems, our approach integrates full-body task-space force controllers with joint-space pattern generators on the commanded joint torque output to facilitate robust control performance, as well as the efficient online learning. With this approach various compliant and stable motions have been created in a constructive manner. We demonstrate the effectiveness of our approach by two folds of experiments: 1) Compliant double / single-support balancing and quasi-static walking on uneven terrain, which do not require any joint patters, 2) Fast and stable squat and dynamic walking by introducing joint-space pattern generators.

In this paper, we propose a novel concept of movement primitives called Stylistic Dynamic Movement Primitives (SDMPs) for motor learning and control in humanoid robotics. In the SDMPs, a diversity of styles in human motion observed through multiple demonstrations can be compactly encoded in a movement primitive, and this allows style manipulation of motion sequences generated from the movement primitive by a control variable called a style parameter. Focusing on discrete movements, a model of the SDMPs is presented as an extension of Dynamic Movement Primitives (DMPs) proposed by Ijspeert et al.. A novel learning procedure of the SDMPs from multiple demonstrations, including a diversity of motion styles, is also described. We present two practical applications of the SDMPs, i.e., stylistic table tennis swings and obstacle avoidance with an anthropomorphic manipulator.

Acquisition of new sensorimotor knowledge by imitation is a promising paradigm for robot learning. To be effective, action learning should not be limited to direct replication of movements obtained during training but must also enable the generation of actions in situations a robot has never encountered before. This paper describes a methodology that enables the generalization of the available sensorimotor knowledge. New actions are synthesized by the application of statistical methods, where the goal and other characteristics of an action are utilized as queries to create a suitable control policy, taking into account the current state of the world. Nonlinear dynamic systems are employed as a motor representation. The proposed approach enables the generation of a wide range of policies without requiring an expert to modify the underlying representations to account for different task-specific features and perceptual feedback. The paper also demonstrates that the proposed methodology can be integrated with an active vision system of a humanoid robot. 3-D vision data are used to provide query points for statistical generalization. While 3-D vision on humanoid robots with complex oculomotor systems is often difficult due to the modeling uncertainties, we show that these uncertainties can be accounted for by the proposed approach.

In this video presentation we introduce a 50 degrees of freedom humanoid robot, CB -ComputationalBrain[1]. CB is a humanoid robot created for exploring the underlying processing of the human brain while dealing with the real world. We place our investigations within real world contexts, as humans do. In so doing, we focus on utilising a system that is closer to humans - in sensing, kinematics configuration and performance. We present a full-body compliance controller that was developed for the motion control of our humanoid robot [2]. Our initial experimentation on our system includes: 1) full-body compliant control - physical interactions/balancing/motion control; 2) the integrated visual ocular-motor responses; 3) perception and control - reaching, foveation, and active object recognition; 4) our studies of Central Pattern Generator for walking.

In this paper, we propose a simple framework for learning and synthesis of fast and complex motor tasks. Where a passivity-based task-space controller acts not only as a full-body force control module, but also as an important module to generate phasic joint patterns. The generated joint patterns are encoded into the parameters of phase oscillators and form the synergy of the task. Then, similar and/or faster motions are synthesized by superposing the task space controller output and the oscillator output with the modified oscillator amplitudes and/or frequencies. We present some examples of whole-body motion synthesis on a human-sized biped humanoid robot including squatting, dancing and stepping while bipedal balancing. The simulation and experimental videos are supplemented.

We propose using the kernel dimension reduction (KDR) to extract a low-dimensional feature space for humanoid locomotion tasks. Although humanoids have many degrees of freedom, task relevant feature spaces can be much smaller than the number of dimension of the original state space. We consider an application of the proposed approach to improve the locomotive performance of humanoid robots using an extracted low-dimensional state space. To improve the locomotive performance, we use a reinforcement learning (RL) framework. While RL is a useful non-linear optimizer, it is usually difficult to apply RL to real robotic systems - due to the large number of iterations required to acquire suitable policies. In this study, we use the extracted low-dimensional feature space for RL so that the learning system can improve task performance quickly. The kernel dimension reduction method allows us to extract the feature space even if the task relevant mapping is non-linear. This is an essential property to improve humanoid locomotive performance since stepping or walking dynamics involves highly nonlinear dynamics. We show that we can improve stepping and walking policies by using a RL method on an extracted feature space by using KDR.

In this paper, we propose an imitation learning framework to generate multiple behaviors with balance control by recognizing human behaviors while estimating the ground reaction force. In our proposed method, a part of captured human motion data is recognized as one particular behavior that is represented by a linear dynamical model. Therefore, our method has small dependence on a classification criteria defined by an experimenter. Based on the behavior recognition method with the ground reaction force estimation and by combining the different recognized behaviors, it is possible to generate many different motion sequences while taking balance into account. First, we approximate a human motion pattern by using linear dynamical models. Then, we can recognize and generate different behavior sequences by switching linear dynamical models. We apply the proposed method to a four-link simulated robot model. Two different squat motions are recognized from motion capture data and the four-link robot generated four different combined squat behaviors from two different squat motions. To show generalization performance, we apply our imitation learning framework to the four-link robot models that have different weights.

Biological systems seem to have a simpler but more robust locomotion strategy than that of the existing biped walking controllers for humanoid robots. We show that a humanoid robot can step and walk using simple sinusoidal desired joint trajectories with their phase adjusted by a coupled oscillator model. We use the center-of-pressure location and velocity to detect the phase of the lateral robot dynamics. This phase information is used to modulate the desired joint trajectories. We do not explicitly use dynamical parameters of the humanoid robot. We hypothesize that a similar mechanism may exist in biological systems. We applied the proposed biologically inspired control strategy to our newly developed human-sized humanoid robot computational brain (CB) and a small size humanoid robot, enabling them to generate successful stepping and walking patterns.

This paper presents a novel approach to acquire dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass. A policy-gradient method is used to acquire a CoM movement as a control policy for achieving a desired dynamic task. A CoM-Jacobian-based redundancy resolution is then used to compute angular velocities for all joints in order to achieve a whole-body movement consistent with the CoM movement acquired through learning. To demonstrate the effectiveness of our method, we apply it in simulation to the learning of a strong punching movement on the Fujitsu humanoid robot, Hoap-2.

We propose to improve the locomotive performance of humanoid robots by using approximated biped stepping and walking dynamics with reinforcement learning (RL). Although RL is a useful non-linear optimizer, it is usually difficult to apply RL to real robotic systems - due to the large number of iterations required to acquire suitable policies. In this study, we first approximated the dynamics by using data from a real robot, and then applied the estimated dynamics in RL in order to improve stepping and walking policies. Gaussian processes were used to approximate the dynamics. By using Gaussian processes, we could estimate a probability distribution of a target function with a given covariance function. Thus, RL can take the uncertainty of the approximated dynamics into account throughout the learning process. We show that we can improve stepping and walking policies by using a RL method with the approximated models both in simulated and real environments. Experimental validation on a real humanoid robot of the proposed

In this study, we propose a novel use of reinforcement learning for estimating hidden variables and parameters of nonlinear dynamical systems. A critical issue in hidden-state estimation is that we cannot directly observe estimation errors. However, by defining errors of observable variables as a delayed penalty, we can apply a reinforcement learning frame-work to state estimation problems. Specifically, we derive a method to construct a nonlinear state estimator by finding an appropriate feedback input gain using the policy gradient method. We tested the proposed method on single pendulum dynamics and show that the joint angle variable could be successfully estimated by observing only the angular velocity, and vice versa. In addition, we show that we could acquire a state estimator for the pendulum swing-up task in which a swing-up controller is also acquired by reinforcement learning simultaneously. Furthermore, we demonstrate that it is possible to estimate the dynamics of the pendulum itself while the hidden variables are estimated in the pendulum swing-up task. Application of the proposed method to a two-linked biped model is also presented.

We show that a humanoid robot can step and walk using simple sinusoidal desired joint trajectories with their phase adjusted by a coupled oscillator model. We use the center of pressure location and velocity to detect the phase of the lateral robot dynamics. This phase information is used to modulate the desired joint trajectories. We applied the proposed control approach to our newly developed human sized humanoid robot and a small size humanoid robot developed by Sony, enabling them to generate successful stepping and walking patterns

Recently, there has been a growing interest in biologically inspired biped locomotion control with Central Pattern Generator (CPG). However, few experimental attempts on real hardware 3D humanoid robots have yet been made. Our goal in this paper is to present our achievement of 3D biped locomotion using a neural oscillator applied to a humanoid robot, QRIO. We employ reduced number of neural oscillators as the CPG model, along with a task space Cartesian coordinate system and utilizing entrainment property to establish stable walking gait. We verify robustness against lateral perturbation, through numerical simulation of stepping motion in place along the lateral plane. We then implemented it on the QRIO. It could successfully cope with unknown 3mm bump by autonomously adjusting its stepping period. Sagittal motion produced by a neural oscillator is introduced, and then overlapped with the lateral motion generator in realizing 3D biped locomotion on a QRIO humanoid robot.

This paper proposes a learning framework for a CPG-based biped locomotion controller using a policy gradient method. Our goal in this study is to develop an efficient learning algorithm by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller in the CPG-based controller can be acquired using the proposed method within a few thousand trials by numerical simulations. Furthermore, we implement the learned controller on the physical biped robot to experimentally show that the learned controller successfully works in the real environment.

We propose a model-based reinforcement learning algorithm for biped walking in which the robot learns to appropriately modulate an observed walking pattern. Via-points are detected from the observed walking trajectories using the minimum jerk criterion. The learning algorithm modulates the via-points as control actions to improve walking trajectories. This decision is based on a learned model of the Poincaré map of the periodic walking pattern. The model maps from a state in the single support phase and the control actions to a state in the next single support phase. We applied this approach to both a simulated robot model and an actual biped robot. We show that successful walking policies are acquired.

This letter proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as modeling errors. The use of environmental models in RL is quite popular for both offline learning using simulations and for online action planning. However, the difference between the model and the real environment can lead to unpredictable, and often unwanted, results. Based on the theory of H(infinity) control, we consider a differential game in which a "disturbing" agent tries to make the worst possible disturbance while a "control" agent tries to make the best control input. The problem is formulated as finding a min-max solution of a value function that takes into account the amount of the reward and the norm of the disturbance. We derive online learning algorithms for estimating the value function and for calculating the worst disturbance and the best control in reference to the value function. We tested the paradigm, which we call robust reinforcement learning (RRL), on the control task of an inverted pendulum. In the linear domain, the policy and the value function learned by online algorithms coincided with those derived analytically by the linear H(infinity) control theory. For a fully nonlinear swing-up task, RRL achieved robust performance with changes in the pendulum weight and friction, while a standard reinforcement learning algorithm could not deal with these changes. We also applied RRL to the cart-pole swing-up task, and a robust swing-up policy was acquired.

Humanoid research has made remarkable progress during the past 10 years. However, currently most humanoids use the target ZMP (zero moment point) control algorithm for bipedal locomotion, which requires precise modeling and actuation with high control gains. On the contrary, humans do not rely on such precise modeling and actuation. Our aim is to examine biologically related algorithms for bipedal locomotion that resemble human-like locomotion. This paper describes an empirical study of a neural oscillator for the control of biped locomotion. We propose a new neural oscillator arrangement applied to a compass-like biped robot. Dynamic simulations and experiments with a real biped robot were carried out and the controller performs steady walking for over 50 steps. Gait variations resulting in energy efficiency was made possible through the adjustment of only a single neural activity parameter. Aspects of adaptability and robustness of our approach are shown by allowing the robot to walk over terrains with varying surfaces with different frictional properties. Initial results suggesting optimal amplitude for dealing with perturbation are also presented.

We propose a model-based reinforcement learning algorithm for biped walking in which the robot learns to appropriately place the swing leg. This decision is based on a learned model of the Poincare map of the periodic walking pattern. The model maps from a state at the middle of a step and foot placement to a state at next middle of a step. We also modify the desired walking cycle frequency based on online measurements. We present simulation results, and are currently implementing this approach on an actual biped robot.

We propose a framework for learning biped locomotion using dynamical movement primitives based on nonlinear oscillators. In our previous work, we suggested dynamical movement primitives as a central pattern generator (CPG) to learn biped locomotion from demonstration. We introduced an adaptation algorithm for the frequency of the oscillators based on phase resetting at the instance of heel strike and entrainment between the phase oscillator and mechanical system using feedback from the environment. In this paper, we empirically explore the role of phase resetting in the proposed algorithm for robust biped locomotion. We demonstrate that phase resetting contributes to robustness against external perturbations and environmental changes by numerical simulations and experiments with a physical biped robot.

We developed a robust control policy design method in high-dimensional state space by using differential dynamic programming with a minimax criterion. As an example, we applied our method to a simulated five link biped robot. The results show lower joint torques from the optimal control policy compared to a hand-tuned PD servo controller. Results also show that the simulated biped robot can successfully walk with unknown disturbances that cause controllers generated by standard differential dynamic programming and the hand-tuned PD servo to fail. Learning to compensate for modeling error and previously unknown disturbances in conjunction with robust control design is also demonstrated. We also applied proposed method to a real biped robot for optimizing swing leg trajectories.

A longstanding goal of reinforcement learning is to develop non- parametric representations of policies and value functions that support rapid learning without suffering from interference or the curse of di- mensionality. We have developed a trajectory-based approach, in which policies and value functions are represented nonparametrically along tra- jectories. These trajectories, policies, and value functions are updated as the value function becomes more accurate or as a model of the task is up- dated. We have applied this approach to periodic tasks such as hopping and walking, which required handling discount factors and discontinu- ities in the task dynamics, and using function approximation to represent value functions at discontinuities. We also describe extensions of the ap- proach to make the policies more robust to modeling error and sensor noise.

We developed a robust control policy design method in high-dimensional state space by using differential dynamic programming with a minimax criterion. As an example, we applied our method to a simulated five link biped robot. The results show lower joint torques from the optimal con- trol policy compared to a hand-tuned PD servo controller. Results also show that the simulated biped robot can successfully walk with unknown disturbances that cause controllers generated by standard differential dy- namic programming and the hand-tuned PD servo to fail. Learning to compensate for modeling error and previously unknown disturbances in conjunction with robust control design is also demonstrated.

This paper proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as mod(cid:173) eling errors. The use of environmental models in RL is quite pop(cid:173) ular for both off-line learning by simulations and for on-line ac(cid:173) tion planning. However, the difference between the model and the real environment can lead to unpredictable, often unwanted results. Based on the theory of H oocontrol, we consider a differential game in which a 'disturbing' agent (disturber) tries to make the worst possible disturbance while a 'control' agent (actor) tries to make the best control input. The problem is formulated as finding a min(cid:173) max solution of a value function that takes into account the norm of the output deviation and the norm of the disturbance. We derive on-line learning algorithms for estimating the value function and for calculating the worst disturbance and the best control in refer(cid:173) ence to the value function. We tested the paradigm, which we call "Robust Reinforcement Learning (RRL)," in the task of inverted pendulum. In the linear domain, the policy and the value func(cid:173) tion learned by the on-line algorithms coincided with those derived analytically by the linear H ootheory. For a fully nonlinear swing(cid:173) up task, the control by RRL achieved robust performance against changes in the pendulum weight and friction while a standard RL control could not deal with such environmental changes.

We propose a learning method for implementing human-like sequential movements in robots. As an example of dynamic sequential movement, we consider the "stand-up" task for a two-joint, three-link robot. In contrast to the case of steady walking or standing, the desired trajectory for such a transient behavior is very difficult to derive. The goal of the task is to find a path that links a lying state to an upright state under the constraints of the system dynamics. The geometry of the robot is such that there is no static solution; the robot has to stand up dynamically utilizing the momentum of its body. We use reinforcement learning, in particular, a continuous time and state temporal difference (TD) learning method. For successful results, we use 1) an efficient method of value function approximation in a high-dimensional state space, and 2) a hierarchical architecture which divides a large state space into a few smaller pieces.