In this study, we present a method to grasp diverse unseen real-world objects using an off-policy actor-critic deep reinforcement learning (RL) with the help of a simulation and the use of as little real-world data as possible. Actor-critic deep RL is unstable and difficult to tune when a raw image is given as an input. Therefore, we use state representation learning (SRL) to make actor-critic RL feasible for visual grasping tasks. Meanwhile, to reduce visual reality gap between simulation and reality, we also employ a typical pixel-level domain adaptation that can map simulated images to realistic ones. In our method, as the SRL model is a common preprocessing module for simulated and real-world data, we perform SRL using real and adapted images. This pixel-level domain adaptation enables the robot to learn grasping skills in a real environment using small amounts of real-world data. However, the controller trained in the simulation should adapt to the real world efficiently. Hence, we propose a method combining a typical pixel-level domain adaptation and the proposed SRL model, where we perform SRL based on a feature-level domain adaptation. In evaluations of vision-based robotics grasping tasks, we show that the proposed method achieves a substantial improvement over a method that only employs a pixel-level or domain adaptation.

For robotic grasping tasks with diverse target objects, some deep learning-based methods have achieved state-of-the-art results using direct visual input. In contrast, actor-critic deep reinforcement learning (RL) methods typically perform very poorly when applied to grasp diverse objects, especially when learning from raw images and sparse rewards. To render these RL techniques feasible for vision-based grasping tasks, we used state representation learning (SRL), in which we encode essential information for subsequent use in RL. However, typical representation learning procedures are unsuitable for extracting pertinent information for learning grasping skills owing to the high complexity of visual inputs for representation learning, in which a robot attempts to grasp a target object. We found that the proposed preprocessed input image is the key to capturing effectively a compact representation. This enables deep RL to learn robotic grasping skills from highly varied and diverse visual inputs. Further, we demonstrate the effectiveness of the proposed approach with varying levels of preprocessing in a realistic simulated environment. We also describe how the resulting model can be transferred to a real-world robot and also demonstrate a 68% success rate on real-world grasp attempts.

For smart manufacturing, an automated robotic assembly system built upon an autoprogramming environment is necessary to reduce setup time and cost for robots that are engaged in frequent task reassignment. This article presents an approach to the autoprogramming of robotic assembly tasks with minimal human assistance. The approach integrates “robotic learning of assembly tasks from observation” and “robotic embodiment of learned assembly tasks in the form of skills.” In the former, robots observe human assembly operations to learn a sequence of assembly tasks, which is formalized into a human assembly script. The latter transforms the human assembly script into a robot assembly script in which a sequence of robot-executable assembly tasks are defined based on action planning supported by workspace modeling and simulated retargeting. The assembly tasks, in the form of the robot assembly script, are then implemented via pretrained robot skills. These skills aim to enable robots to execute difficult tasks that involve inherent uncertainties and variations. We validate the proposed approach by building a prototype of the automated robotic assembly system for a power breaker and an electronic set-top box. The results verify that the proposed automated robotic assembly system is not only feasible but also viable, as it is associated with a dramatic reduction in the human effort required for automating robotic assembly.

In this paper, we propose a method to model social interaction between a human and a virtual avatar. To this end, two human performers fist perform social interactions according to the Learning from Demonstration paradigm. Then, the relative relevance of all joints of both performers should be reasonably modeled based on human demonstrations. However, among all possible combinations of relative joints, it is necessary to select only some of the combinations that play key roles in social interaction. We select such significant features based on the joint motion significance, which is a metric to measure the significance degree by calculating both temporal entropy and spatial entropy of all human joints from a Gaussian mixture model. To evaluate our proposed method, we performed experiments on five social interactions: hand shaking, hand slapping, shoulder holding, object passing, and target kicking. In addition, we compared our method to existing modeling methods using different metrics, such as principal component analysis and information gain.

In this paper, we propose an approach for a robot to adapt time scale of motions to guarantee temporal constraints of robot tasks. To this end, an selective Gaussian mixture model is first presented based on motion significance. Also, an selective Gaussian mixture regression is then presented to select time indexes for regression of robot motions from Gaussian mixtures based on motion significance. Motion significance indicates the relative significance of every motion frame, which is defined as a set of data points acquired at a time index of multiple motion trajectories. Such motion significance is measured by combining both temporal entropy and spatial entropy of a motion frame, based on the analysis of Gaussian mixtures. To evaluate our proposed method, we perform the experiment in which a robot draws a figure on a whiteboard. In addition, we discuss how to develop the adaptive time scaling by adjusting piecewise Gaussian distributions involved in Gaussian mixtures.

In this paper, we propose a novel method for learning motor skills based on grossness and fineness of movements involved in daily-life tasks. Grossness and fineness depend on the degrees of complexity (i.e., linear combinations between basis vectors) and repeatability (i.e., repeat accuracies between multiple trials) of such movements. In such a daily-life task, a robot's movements are usually related to a task-relevant object. Therefore, the complexity and the repeatability should be acquired from datasets that include the spatial and temporal relationships between a robot and a task-relevant object. To measure the degree of complexity, correlations are first obtained from each data by canonical correlation analysis. To measure the degree of repeatability, variations are then obtained from covariances between datasets acquired by multiple trials. The grossness and fineness are finally acquired by combining the correlations and the variations. To learn a motor skill, a Gaussian Mixture Model (GMM) is estimated using well-known methods as Principal Component Analysis (PCA), k-means, Bayesian Information Criterion (BIC), and Expectation-Maximization (EM) algorithms. First, initial parameters of a GMM are estimated by weighting a conventional k-means algorithm with the grossness and fineness. Based on PCA, BIC, and EM algorithms, the GMM is then estimated using the initial parameters and a robot's motion trajectories. To validate our proposed methods, the GMM is evaluated in terms of reproduction and recognition using a robot arm that performs two daily-life tasks: cookie-decorating and constrained-delivering tasks.

A Layered Hidden Markov Model (LHMM) has been usually used for recognizing various human activities. In such a LHMM, the performance tends to be improved than that of a single layered HMM. To further enhance the performance of such a LHMM, in this paper, we propose a brain-inspired feedback mechanism. For this achievement, the LHMM is first modeled using a set of training data that the semantic information (i.e., labels of data) is attached. In the inference phase, the semantic information is produced from the HMMs associated with the upper layers of the LHMM, and then the semantic information is used to improve the performances of the lower layers in the next inference step. Consequently, these interactive feed-forward and feedback information can dramatically improve the performance of the LHMM. To validate our proposed method, we compare the performance of our LHMM (i.e., with feedback mechanism) with that of a standard LHMM (i.e., with no feedback mechanism) using twenty-four human activities, which occur frequently when a human cooks.

In manipulation tasks, skills are usually modeled using the continuous motion trajectories acquired in the task space. The motion trajectories obtained from a human's multiple demonstrations can be broadly divided into four portions, according to the spatial variations between the demonstrations and the time spent in the demonstrations: the portions in which a long/short time is spent, and those in which the spatial variations are large/small. In these four portions, the portions in which a long time is spent and the spatial variation is small (e.g., passing a thread through the eye of a needle) are usually modeled using a small number of parameters, even if such portions represent the movement that is essential for achieving the task. The reason for this is that these portions are slightly changed in the task space as compared with the other portions. In fact, such portions should be densely modeled using more parameters (i.e., overfitting) to improve the performance of the skill because the movements of those portions must be accurately executed to achieve the task. In this paper, we propose a method for adaptively fitting these skills based on the temporal and the spatial entropies calculated by a Gaussian mixture model. We found that it is possible to retrieve accurate motion trajectories as compared with those of well-fitted models, whereas the estimation performance is generally higher than that of an overfitted model. To validate our proposed method, we present the experimental results and evaluations when using a robot arm that performed two tasks.

To achieve a certain task, a skilligent robot should be able to learn the skills embedded in that task. Furthermore, the robot should be able to infer such skills to handle uncertainties and perturbations, since most robot tasks are usually daily-life tasks that include many unexpected situations. Therefore, we propose a unified skill learning and inference framework. The framework includes six processing modules: 1) a human demonstration process, 2) an autonomous segmentation process, 3) a dynamic movement primitive learning process, 4) a Bayesian network learning process, 5) a motivation graph construction process, and 6) a skill-inferring process. Based on the framework, the robot learns and infers situation-adequate and goal-oriented skills to handle uncertainties and human perturbations. To show the validity of our framework, some experimental results are illustrated using a robot arm that performs a `tea service' task.

In this work, we propose methods for automatically generating primitive skills from demonstration of a task. Additionally, we propose methods for improving the existing primitive skills and adding the new primitive skills incrementally and automatically. To validate our proposed methods, we present experimental results of a human-like robot handling three gestures and a task for making coffee.

We first generate a probabilistic affordance to select an action based on motivation values. The affordance is designed as a multilayer naïve Bayesian classifier with respect to uncertainties and equivalence classes. The multilayer naïve Bayesian classifier is a probabilistic model with multiple layers of conditional probability tables and/or probability distributions to represent the equivalence classes. The affordances are arranged based on goal-orientedness, since achieving a task usually requires actions performed in a sequence. Additionally, motivation values are generated using the arranged affordances and a motivation value propagation algorithm. A robot selects a goal-oriented as well as a situation-adequate action based on the motivation values. To validate our proposed methods, we present experimental results of an entertainment robot called AIBO, handling three tasks.

A Skilligent robot must be able to learn skills autonomously to accomplish a task. ldquoskilligencerdquo is the capacity of the robot to control behaviors reasonably, based on the skills acquired during run-time. Behavior control based on Bayesian networks is used to control reasonable behaviors. To accomplish this, subgoals are first discovered by clustering similar features of state transition tuples, which are composed of current states, actions, and next states. Here, features used in clustering are produced using changes of the states in the state transition tuples. Parameters of Bayesian networks and utility functions are learned separately using state transition tuples belonging to each subgoal. To select the best action while executing a task, the expected utility of each subgoal is calculated by the expected utility function and the robot chooses the action that maximizes expected utility calculated by the maximum expected utility (MEU) function. The MEU function is based on the conditional probabilistic distributions of Bayesian networks and utility functions. We also propose a method for reconstructing learned networks and increasing subgoals by incremental learning. To show the validities of our proposed methods, a task using dribbling-box-into-a-goal (DBIG) and obstacle-avoidance-while-dribbling-box (OAWDB) skills is simulated and experimented.