Recently, considerable research has been focused on multi-agent reinforcement learning to effectively account for each agent's relations. However, most research has focused on homogeneous multi-agent systems with the same type of agents, which has limited application to heterogeneous multi-agent systems. The demand for heterogeneous systems has considerably increased not only in games but also in the real world. Therefore, a technique that can properly consider relations in heterogeneous systems is required. In this study, we propose a novel transformer network calledHRformer, which is based on heterogeneous graph networks that can reflect the heterogeneity and relations among agents. To this end, we design an effective linear encoding method for the transformer to receive input of the various and unique characteristics of the agents and introduce a novel encoding method to model the relations among them. Experiments are conducted in the StarCraft Multi-Agent Challenge (SMAC) environment, the most famous heterogeneous multi-agent simulation, to demonstrate the superior performance of the proposed method compared to other existing methods in various heterogeneous scenarios. The proposed method in our simulation shows a high win rate and fast convergence speed, proving the superiority of the proposed method considering the heterogeneity of the multi-agent system.

While Generative Adversarial Imitation Learning (GAIL) shows remarkable performance in many high dimensional imitation learning tasks, it requires too many sampled transitions, which are infeasible for some real world problems. In this paper, we demonstrate how exploiting the reward function in GAIL can improve sample efficiency. We design our algorithm end-to-end differentiable so that the learned reward function can directly participate in policy updates. End-to-end differentiability can be achieved by introducing a forward model of the environment, enabling direct calculation of the cumulative reward function. However, using a forward model has two significant limitations that it heavily relies on the performance of the forward model and requires multi-step prediction, which causes severe error accumulation. The proposed end-to-end differentiable adversarial imitation learning algorithm alleviates these limitations. Also, we suggest applying several existing regularization techniques for robust training of a forward model. We call our algorithm, integrated with these regularization methods, fully Differentiable Regularized GAIL (DRGAIL), and test DRGAIL on continuous control tasks.

In order to handle complex navigational commands, this paper proposes a novel algorithm that can modify a walking period and a step length in both sagittal and lateral planes. By allowing a variation of zero moment point (ZMP) over the convex hull of foot polygon, it is possible to change the center of mass (CM) position and velocity independently throughout the single support phase. This permits a range of dynamic walking motion, which is not achievable using the 3-D linear inverted pendulum mode (3D-LIPM). In addition, the proposed algorithm enables to determine the dynamic feasibility of desired motion via the construction of feasible region, which is explicitly computed from the current CM state with simple ZMP functions. Moreover, adopting the closed-form functions makes it possible to calculate the algorithm in real time. The effectiveness of the proposed algorithm is demonstrated through both computer simulation and experiment on the humanoid robot, HanSaRam-VII, developed at the Robot Intelligence Technology (RIT) laboratory, Korea Advanced Institute of Science and Technology (KAIST).

This paper presents the components and overall architecture of the ubiquitous robot (Ubibot) system developed to demonstrate ubiquitous robotics, a new paradigm for integrated services. The system has been developed on the basis of the definition of the ubiquitous robot as that of encompassing the software robot Sobot, embedded robot Embot and the mobile robot Mobot. This tripartite partition, which independently manifests intelligence, perception and action, enables the abstraction of intelligence through the standardization of sensory data and motor or action commands. The Ubibot system itself is introduced along with its component subsystems of Embots, the position Embot, vision Embot and sound Embot, the Mobots of Mybot and HSR, the Sobot, Rity, a virtual pet modeled as an artificial creature, and finally the middleware which seamlessly enables interconnection between other components. Three kinds of experiments are devised to demonstrate the fundamental features, of calm sensing, context awareness and seamless service transcending the spatial limitations in the abilities of earlier generation personal robots. The experiments demonstrate the proof of concept of this powerful new paradigm which shows great promise.

Complex navigational commands require a walk ing pattern generator that is able to modify the pattern at any point in the walking gait. This paper utilizes the 3D-LIPM(Linear Inverted Pendulum Model) for generating a walking pattern, but introduces a method that allows for manipulation of the ZMP over the convex hull of the foot polygon whilst in single support phase. This permits a range of dynamic walking states that are not achievable using the conventional 3D-LIPM. These walking states are defined as a feasible region. A real-time algorithm is then developed, which follows the cue from a complex navigational command exactly when the desired walking state is in the feasible region and chooses the nearest feasible motion when it lies outside the feasible region. The proposed scheme is both simulated and implemented on the humanoid robot HanSaRam-VII developed at RIT laboratory, KAIST.

This paper presents the development of generating dance performances of humanoid robot, HSR (HanSaRam)-VII, synchronized with Robonova, for entertainment. This heterogeneous team, RoboBees, participated in 'Robots at Play Award 2006' and nominated as one of top six teams in the world. A method of generating and combining both periodic motion (on-line pattern generator) and aperiodic motion (off-line pattern generator) is investigated along with the control architecture for HSR-VII. Detailed system descriptions of two types of humanoid robots, HSR-VII from KAIST and Robonova from miniROBOT Corp., are included with the experimental results. In addition, the time domain passivity compliance control system is introduced to guarantee the stable periodic motion by changing the initial planned trajectories. Video clip of RoboBees' performance is available at http://rit.kaist.ac.kr.

This paper proposes a control method to absorb the landing force or the ground reaction force for a stable dynamic walking of a humanoid robot. Humanoid robot may become unstable during walking due to the impulsive contact force of the sudden landing of its foot. Therefore, a control method to decrease the landing force is required. In this paper, time-domain passivity control approach is applied for this purpose. Ground and the foot of the robot are modeled as two one-port network systems that are connected, and exchange energy with each other. The time-domain passivity controller with admittance causality is implemented, which has the landing force as input and foot's position to trim off the force as output. The proposed landing force controller can enhance the stability of the walking robot from simple computation. The small-sized humanoid robot, HanSaRam-VII that has 27 DOFs, is developed to verify the proposed scheme through dynamic walking experiments.

This paper proposes a genetic representation method to evolve artificial creature’s personality by using artificial genome and evolutionary generative algorithm. The genome consists of computer-coded chromosomes. Based on the internal architecture, the chromosomes are designed for the genetic representation. They are composed of the fundamental genes, internal state related genes, and behavior related genes as essential components, which represent the personality and are used for the animal-like evolution in the simulated environment. To get a desired personality (genome), a proper fitness function is designed and genetic operators are applied to the population (genomes). The artificial creature, Rity, is developed in a virtual world of PC to test the effectiveness of this representation scheme.

In this paper, a method to reduce the landing impact force is proposed for a stable dynamic walking of a humanoid robot. To measure the meaningful landing impact force, a novel foot mechanism, which uses FSRs (force sensing resistors), is introduced as well. Humanoid robot might become unstable during the walking due to the impulsive contact force from the sudden landing of its foot. Therefore a new control method to decrease the landing impact force has been required. In this paper, time domain passivity control approach is applied for this purpose. Ground and the foot of the robot are modeled as two one-port network systems which are connected and exchanging energy each other. And, the time domain passivity controller which has the landing impact force as input and foot's position to trim off the force as output, is implemented. Unlike previous works, the proposed controller can guarantee the stability of the robot system without any dynamic model information at all. The small sized humanoid robot, HanSaRam-VI which has 25 DOFs, with the proposed foot mechanism is developed to verify the proposed approach through dynamic walking experiments

For the purpose of a humanoid robot's stable walking or running, it is important to absorb landing force or ground reaction force which is generated when the robot's foot lands on the ground surface. The force can make the robot unstable, and the problem becomes serious if the robot runs. This paper proposes a control system, which can absorb the landing force of a humanoid robot. Time-domain passivity control approach is applied for this purpose. Ground and the robot's foot are modeled as two one-port network systems, which are connected and exchange energy with each other. The time-domain passivity controller has the landing force as input and controls the foot's position to reduce the force. The proposed controller can guarantee the stability of the robot system without need of any dynamic model information or control parameters. Using small sized humanoid robot, dynamic walking experiments are performed to verify the proposed scheme, and its efficiency is shown from the comparison with the other scheme.

This paper proposes a behavior selection mechanism and interactive learning method for a synthetic character. Synthetic character can decide its behavior by itself based on its own internal states (motivation, homeostasis, and emotion), and external sensor information. A behavior is selected by both probabilistic and deterministic methods. The probabilistic method uses the internal states and external sensor information, and the deterministic method which imitates animal's instinct, uses only external sensor information. Both methods are complementary to each other. A user can teach the synthetic character a desired behavior among many behaviors, and behaviors are grouped into analogous behavior sets. The learning algorithm includes the emotional parameters by which the training efficiency is affected. The performance of the synthetic character, Rity, developed with the proposed mechanism at RIT Lab., KAIST is demonstrated in a 3D virtual world.