The current study investigated possible human-robot kinaesthetic interactions using a variational recurrent neural network (RNN) model, called PV-RNN, which is based on the free-energy principle. Our prior robotic studies using PV-RNN showed that the nature of interactions between the top-down expectation and bottom-up inference is strongly affected by a parameter, called the meta-prior, which regulates the complexity term in free energy. The current study examines how changing the meta-prior w in the interaction phase affects the counter force generated when an experimenter attempts to induce movement pattern transitions familiar to the robot through its prior training. The study also compares the counter force generated when trained transitions are induced by a human experimenter and when untrained transitions are induced. Finally, the study examines how different levels of disturbances, pattern level, and cognitive level can be resolved internally across different layers. Our experimental results indicated that 1) the human experimenter needs more/less force to induce trained transitions when w is set with larger/smaller values; 2) the human experimenter needs more force to act on the robot when he attempts to induce untrained as opposed to trained movement pattern transitions; and 3) when the robot was disturbed in the cognitive level, the disturbance was resolved more in the higher layer as compared to the case with disturbance in the pattern level. Our analysis of time development of essential variables and values in PV-RNN during bodily interaction clarified the mechanism by which gaps in action intentions between the human experimenter and the robot can be manifested as reaction forces between them.

In reinforcement learning (RL), artificial agents are trained to maximize numerical rewards by performing tasks. Exploration is essential in RL because agents must discover information before exploiting it. Two rewards encouraging efficient exploration are the entropy of action policy and curiosity for information gain. Entropy is well established in the literature, promoting randomized action selection. Curiosity is defined in a broad variety of ways in literature, promoting discovery of novel experiences. One example, prediction error curiosity, rewards agents for discovering observations they cannot accurately predict. However, such agents may be distracted by unpredictable observational noises known as curiosity traps. Based on the free energy principle (FEP), this letter proposes hidden state curiosity, which rewards agents by the KL divergence between the predictive prior and posterior probabilities of latent variables. We trained six types of agents to navigate mazes: baseline agents without rewards for entropy or curiosity and agents rewarded for entropy and/or either prediction error curiosity or hidden state curiosity. We find that entropy and curiosity result in efficient exploration, especially both employed together. Notably, agents with hidden state curiosity demonstrate resilience against curiosity traps, which hinder agents with prediction error curiosity. This suggests implementing the FEP that may enhance the robustness and generalization of RL models, potentially aligning the learning processes of artificial and biological agents.

Evolution and development operate at different timescales; generations for the one, a lifetime for the other. These two processes, the basis of much of life on earth, interact in many non-trivial ways, but their temporal hierarchy-evolution overarching development-is observed for most multicellular life forms. When designing robots, however, this tenet lifts: It becomes-however natural-a design choice. We propose to inverse this temporal hierarchy and design a developmental process happening at the phylogenetic timescale. Over a classic evolutionary search aimed at finding good gaits for tentacle 2D robots, we add a developmental process over the robots' morphologies. Within a generation, the morphology of the robots does not change. But from one generation to the next, the morphology develops. Much like we become bigger, stronger, and heavier as we age, our robots are bigger, stronger, and heavier with each passing generation. Our robots start with baby morphologies, and a few thousand generations later, end-up with adult ones. We show that this produces better and qualitatively different gaits than an evolutionary search with only adult robots, and that it prevents premature convergence by fostering exploration. In addition, we validate our method on voxel lattice 3D robots from the literature and compare it to a recent evolutionary developmental approach. Our method is conceptually simple, and it can be effective on small or large populations of robots, and intrinsic to the robot and its morphology, not the task or environment. Furthermore, by recasting the evolutionary search as a learning process, these results can be viewed in the context of developmental learning robotics.

Generalization by learning is an essential cognitive competency for humans. For example, we can manipulate even unfamiliar objects and can generate mental images before enacting a preplan. How is this possible? Our study investigated this problem by revisiting our previous study (Jung, Matsumoto, & Tani, 2019), which examined the problem of vision-based, goal-directed planning by robots performing a task of block stacking. By extending the previous study, our work introduces a large network comprising dynamically interacting submodules, including visual working memory (VWMs), a visual attention module, and an executive network. The executive network predicts motor signals, visual images, and various controls for attention, as well as masking of visual information. The most significant difference from the previous study is that our current model contains an additional VWM. The entire network is trained by using predictive coding and an optimal visuomotor plan to achieve a given goal state is inferred using active inference. Results indicate that our current model performs significantly better than that used in Jung et al. (2019), especially when manipulating blocks with unlearned colors and textures. Simulation results revealed that the observed generalization was achieved because content-agnostic information processing developed through synergistic interaction between the second VWM and other modules during the course of learning, in which memorizing image contents and transforming them are dissociated. This letter verifies this claim by conducting both qualitative and quantitative analysis of simulation results.

Embodiment and subjective experience in humanrobot interaction are important aspects to consider when studying both natural cognition and adaptive robotics to human environments. Although several researches have focused on nonverbal communication and collaboration, the study of autonomous physical interaction has obtained less attention. From the perspective of neurorobotics, we investigate the relation between intentionality, motor compliance, cognitive compliance, and behavior emergence. We propose a variational model inspired by the principles of predictive coding and active inference to study intentionality and cognitive compliance, and an intermittent control concept for motor deliberation and compliance based on torque feed-back. Our experiments with the humanoid Torobo portrait interesting perspectives for the bio-inspired study of developmental and social processes.

Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.

This paper aims to investigate how adequate cognitive functions for recognizing, predicting, and generating a variety of actions can be developed through iterative learning of action-caused dynamic perceptual patterns. Particularly, we examined the capabilities of mental simulation of one's own actions as well as the inference of others' intention because they play a crucial role, especially in social cognition. We propose a dynamic neural network model based on predictive coding which can generate and recognize dynamic visuo-proprioceptive patterns. The proposed model was examined by conducting a set of robotic simulation experiments in which a robot was trained to imitate visually perceived gesture patterns of human subjects in a simulation environment. The experimental results showed that the proposed model was able to develop a predictive model of imitative interaction through iterative learning of large-scale spatio-temporal patterns in visuo-proprioceptive input streams. Also, the experiment verified that the model was able to generate mental imagery of dynamic visuo-proprioceptive patterns without feeding the external inputs. Furthermore, the model was able to recognize the intention of others by minimizing prediction error in the observations of the others' action patterns in an online manner. These findings suggest that the error minimization principle in predictive coding could provide a primal account for the mirror neuron functions for generating actions as well as recognizing those generated by others in a social cognitive context.

This paper introduces our approach to building a robot with communication capability based on the two key features: stochastic neural dynamics and prediction error minimization. A preliminary experiment showed that the humanoid robot was able to imitate other's action by means of those key features. In addition, we found that some sorts of communicative patterns emerged between two robots in which the robots inferred the intention of another agent behind the sensory observation.

Mental simulation is a critical cognitive function for goal-directed behavior because it is essential for assessing actions and their consequences. When a self-generated or externally specified goal is given, a sequence of actions that is most likely to attain that goal is selected among other candidates via mental simulation. Therefore, better mental simulation leads to better goal-directed action planning. However, developing a mental simulation model is challenging because it requires knowledge of self and the environment. The current paper studies how adequate goal-directed action plans of robots can be mentally generated by dynamically organizing top-down visual attention and visual working memory. For this purpose, we propose a neural network model based on variational Bayes predictive coding, where goal-directed action planning is formulated by Bayesian inference of latent intentional space. Our experimental results showed that cognitively meaningful competencies, such as autonomous top-down attention to the robot end effector (its hand) as well as dynamic organization of occlusion-free visual working memory, emerged. Furthermore, our analysis of comparative experiments indicated that the introduction of visual working memory and the inference mechanism using variational Bayes predictive coding significantly improved the performance in planning adequate goal-directed actions.

This study introduces PV-RNN, a novel variational RNN inspired by predictive-coding ideas. The model learns to extract the probabilistic structures hidden in fluctuating temporal patterns by dynamically changing the stochasticity of its latent states. Its architecture attempts to address two major concerns of variational Bayes RNNs: how latent variables can learn meaningful representations and how the inference model can transfer future observations to the latent variables. PV-RNN does both by introducing adaptive vectors mirroring the training data, whose values can then be adapted differently during evaluation. Moreover, prediction errors during backpropagation-rather than external inputs during the forward computation-are used to convey information to the network about the external data. For testing, we introduce error regression for predicting unseen sequences as inspired by predictive coding that leverages those mechanisms. As in other variational Bayes RNNs, our model learns by maximizing a lower bound on the marginal likelihood of the sequential data, which is composed of two terms: the negative of the expectation of prediction errors and the negative of the Kullback-Leibler divergence between the prior and the approximate posterior distributions. The model introduces a weighting parameter, the meta-prior, to balance the optimization pressure placed on those two terms. We test the model on two data sets with probabilistic structures and show that with high values of the meta-prior, the network develops deterministic chaos through which the randomness of the data is imitated. For low values, the model behaves as a random process. The network performs best on intermediate values and is able to capture the latent probabilistic structure with good generalization. Analyzing the meta-prior's impact on the network allows us to precisely study the theoretical value and practical benefits of incorporating stochastic dynamics in our model. We demonstrate better prediction performance on a robot imitation task with our model using error regression compared to a standard variational Bayes model lacking such a procedure.

This letter proposes a novel predictive coding type neural network model, the predictive multiple spatiotemporal scales recurrent neural network (P-MSTRNN). The P-MSTRNN learns to predict visually perceived human whole-body cyclic movement patterns by exploiting multiscale spatiotemporal constraints imposed on network dynamics by using differently sized receptive fields as well as different time constant values for each layer. After learning, the network can imitate target movement patterns by inferring or recognizing corresponding intentions by means of the regression of prediction error. Results show that the network can develop a functional hierarchy by developing a different type of dynamic structure at each layer. The letter examines how model performance during pattern generation, as well as predictive imitation, varies depending on the stage of learning. The number of limit cycle attractors corresponding to target movement patterns increases as learning proceeds. Transient dynamics developing early in the learning process successfully perform pattern generation and predictive imitation tasks. The letter concludes that exploitation of transient dynamics facilitates successful task performance during early learning periods.

The current paper presents a novel recurrent neural network model, predictive multiple spatio-temporal scales RNN (P-MSTRNN), which can generate as well as recognize dynamic visual patterns in a predictive coding framework. The model is characterized by multiple spatio-temporal scales imposed on neural unit dynamics through which an adequate spatio-temporal hierarchy develops via learning from exemplars. The model was evaluated by conducting an experiment of learning a set of whole body human movement patterns, which was generated by following a hierarchically defined movement syntax. The analysis of the trained model clarifies what types of spatio-temporal hierarchy develops in dynamic neural activity as well as how robust generation and recognition of movement patterns can be achieved by using the error minimization principle.

Lifelong learning is fundamental in autonomous robotics for the acquisition and fine-tuning of knowledge through experience. However, conventional deep neural models for action recognition from videos do not account for lifelong learning but rather learn a batch of training data with a predefined number of action classes and samples. Thus, there is the need to develop learning systems with the ability to incrementally process available perceptual cues and to adapt their responses over time. We propose a self-organizing neural architecture for incrementally learning to classify human actions from video sequences. The architecture comprises growing self-organizing networks equipped with recurrent neurons for processing time-varying patterns. We use a set of hierarchically arranged recurrent networks for the unsupervised learning of action representations with increasingly large spatiotemporal receptive fields. Lifelong learning is achieved in terms of prediction-driven neural dynamics in which the growth and the adaptation of the recurrent networks are driven by their capability to reconstruct temporally ordered input sequences. Experimental results on a classification task using two action benchmark datasets show that our model is competitive with state-of-the-art methods for batch learning also when a significant number of sample labels are missing or corrupted during training sessions. Additional experiments show the ability of our model to adapt to non-stationary input avoiding catastrophic interference.

We suggest that different behavior generation schemes, such as sensory reflex behavior and intentional proactive behavior, can be developed by a newly proposed dynamic neural network model, named stochastic multiple timescale recurrent neural network (S-MTRNN). The model learns to predict subsequent sensory inputs, generating both their means and their uncertainty levels in terms of variance (or inverse precision) by utilizing its multiple timescale property. This model was employed in robotics learning experiments in which one robot controlled by the S-MTRNN was required to interact with another robot under the condition of uncertainty about the other's behavior. The experimental results show that self-organized and sensory reflex behavior-based on probabilistic prediction-emerges when learning proceeds without a precise specification of initial conditions. In contrast, intentional proactive behavior with deterministic predictions emerges when precise initial conditions are available. The results also showed that, in situations where unanticipated behavior of the other robot was perceived, the behavioral context was revised adequately by adaptation of the internal neural dynamics to respond to sensory inputs during sensory reflex behavior generation. On the other hand, during intentional proactive behavior generation, an error regression scheme by which the internal neural activity was modified in the direction of minimizing prediction errors was needed for adequately revising the behavioral context. These results indicate that two different ways of treating uncertainty about perceptual events in learning, namely, probabilistic modeling and deterministic modeling, contribute to the development of different dynamic neuronal structures governing the two types of behavior generation schemes.

It is undeniable that the ability to grasp and handle an object is vital for service robots. From object recognition to object grasping motion, the motion execution should be as fast as possible. Due to the possible position variation of the target object to be grasped, online planning of grasping motion should be done. In order to achieve flexible grasping motion, recurrent neural network could be implemented as an alternative to conventional manipulation method which is based on kinematic and dynamic analysis. However, the application of recurrent neural network model requires good and easily obtainable training data. Hence, a novel robotic arm design with high flexibility is proposed to facilitate the training and implementation of the recurrent neural network model. The feasibility of the proposed robotic arm design is evaluated via the training, learning and testing of stochastic continuous time recurrent neural network (S-CTRNN) model with grasping a box motion.

The most pressing question about cognitive brains is how they support the compositionality that enables combinatorial manipulations of images, thoughts, and actions. When addressing this problem with synthetic modeling, the conventional idea prevalent in artificial intelligence and cognitive science, generally, is to assume hybrid systems and corresponding neural network models, where higher order cognition is realized by means of symbolic representation and lower sensory-motor processes by analog processing. However, the crucial problem with such approaches is that the symbols represented at higher order cognitive levels cannot be grounded naturally in sensory-motor reality. The former are defined in a discrete space without any metric, and the latter are defined in a continuous space with a physical metric. These, therefore, cannot directly interact with each other, regardless of the interface that is assigned between them. The proposal in the current paper is to reconstruct higher order cognition by means of continuous neurodynamic systems that can elaborate delicate interactions with the sensory-motor level while sharing the same metric space. Our neurorobotics experiments-including language-action associations and the learning of goal-directed actions-show that the compositionality necessary for higher order cognitive tasks can be acquired by means of self-organizing dynamic structures, via interactive learning between the top-down intentional process of acting on the physical world and the bottom-up recognition of perceptual reality. Using robotic simulations, the current paper demonstrates that nonlinear dynamic phenomena, such as bifurcations and the chaotic dynamics induced by unstable fixed points, could play essential roles in realizing higher order functions.

The foremost objective of our research series is to construct a neurocomputational model that aims to achieve a Large-Scale Brain Network, and to suggest a possible insight of how the macro-level anatomical structures, such as the connectivity between the frontal lobe regions and their dynamic properties, can be self-organized to obtain the higher-order cognitive mechanisms, such as: planning, reasoning, task switching, cognitive branching, etc. For addressing these issues, this paper, in particular, focuses in proposing a model that intends to clarify the neural structure and mechanisms underlying the task switching and the cognitive branching condition. Although both tasks requiring varying degree of a working memory, in contrast to the switching task, where the primary ongoing task is entirely replaced by a new task, in the branching task, a delaying to the execution of an original task occurs until the completion of a subordinate task. The proposed model is constructed by a hierarchical Multiple Timescale Recurrent Neural Network (MTRNN) and conducted on a humanoid robot in a physical environment. Experimental results suggest essential factors related to the neural activities and network's structure necessary to form a suitable working memory for accomplishing such tasks.

This paper proposes a novel network model for estimation of interaction intensity among partially observed signals. The network model can acquire connectivity weights among the signals as a forward model through iterative learning using past and future signals. To evaluate accuracy of the estimation, the model was applied on artificial and physiological data. In case of artificial signals, when all signals were used, the network was able to estimate directional interactions. On the other hand, the network failed to estimate directional interactions when only parts of the signals were used. However, the network was able to estimate whether interactions exist, and signals were successfully grouped into each of its sources using the obtained connectivity. Furthermore, for physiological signals, we obtained connectivity weights that cluster the recording electrode sites into physiologically plausible brain areas. These results suggest that the proposed network model can be used to estimate the clustered interactions from the partially observed signals.

Affordance theory suggests that humans recognize the environment based on invariants. Invariants are features that describe the environment offering behavioral information to humans. Two types of invariants exist, structural invariants and transformational invariants. In our previous paper, we developed a method that self-organizes transformational invariants, or motion features, from camera images based on robot's experiences. The model used a bi-directional technique combining a recurrent neural network for dynamics learning, namely Recurrent Neural Network with Parametric Bias (RNNPB), and a hierarchical neural network for feature extraction. The bi-directional training method developed in the previous work was effective in clustering the motion of objects, but the analysis did not give good segregation results of the self-organized features (transformational invariants) among different motion types. In this paper, we present a refined model which integrates dynamics learning and feature extraction in a single model. The refined model is comprised of Multiple Timescales Recurrent Neural Network (MTRNN), which possesses better learning capability than RNNPB. Self-organization result of four types of motions have proved the model's capability to create clusters of object motions. The analysis showed that the model extracted feature sequences with different characteristics for four object motion types.

Humans are said to unintentionally trace handwriting sequences in their brains based on handwriting experiences when recognizing written text. In this paper, we propose a model for predicting handwriting sequence for written text recognition based on handwriting experiences. The model is first trained using image sequences acquired while writing text. The image features of sequences are self-organized from the images using Self-Organizing Map. The feature sequences are used to train a neuro-dynamics learning model. For recognition, the text image is input into the model for predicting the handwriting sequence and recognition of the text. We conducted two experiments using ten Japanese characters. The results of the experiments show the effectivity of the model.

The current article suggests that deterministic chaos self-organized in cortical dynamics could be responsible for the generation of spontaneous action sequences. Recently, various psychological observations have suggested that humans and primates can learn to extract statistical structures hidden in perceptual sequences experienced during active environmental interactions. Although it has been suggested that such statistical structures involve chunking or compositional primitives, their neuronal implementations in brains have not yet been clarified. Therefore, to reconstruct the phenomena, synthetic neuro-robotics experiments were conducted by using a neural network model, which is characterized by a generative model with intentional states and its multiple timescales dynamics. The experimental results showed that the robot successfully learned to imitate tutored behavioral sequence patterns by extracting the underlying transition probability among primitive actions. An analysis revealed that a set of primitive action patterns was embedded in the fast dynamics part, and the chaotic dynamics of spontaneously sequencing these action primitive patterns was structured in the slow dynamics part, provided that the timescale was adequately set for each part. It was also shown that self-organization of this type of functional hierarchy ensured robust action generation by the robot in its interactions with a noisy environment. This article discusses the correspondence of the synthetic experiments with the known hierarchy of the prefrontal cortex, the supplementary motor area, and the primary motor cortex for action generation. We speculate that deterministic dynamical structures organized in the prefrontal cortex could be essential because they can account for the generation of both intentional behaviors of fixed action sequences and spontaneous behaviors of pseudo-stochastic action sequences by the same mechanism.

Executive control incorporates cognitive functions involved in the control and management of other cognitive processes. Such high-level skills are hard to be explored with brain imaging studies because they require complex and persistent experimental procedures. Alternatively, computational modeling may provide a new way to indirectly explore executive control mechanisms. The current work adopts this latter approach to explore possible characteristics of executive control, focusing particularly on behavioral rule switching and confidence neurodynamics in artificial agents. To this end, our study explores a robotic version of the classical Wisconsin Card Sorting Test, incorporating also the option of betting. Our ability to perform multiple and statistically independent computational experiments together with the in-depth study of the mechanisms created in the artificial cognitive systems, provides suggestions for the executive control aspects of the human brain.

Predictability is an important factor for determining robot motions. This paper presents a model to generate robot motions based on reliable predictability evaluated through a dynamics learning model which self-organizes object features. The model is composed of a dynamics learning module, namely Recurrent Neural Network with Parametric Bias (RNNPB), and a hierarchical neural network as a feature extraction module. The model inputs raw object images and robot motions. Through bi-directional training of the two models, object features which describe the object motion are self-organized in the output of the hierarchical neural network, which is linked to the input of RNNPB. After training, the model searches for the robot motion with high reliable predictability of object motion. Experiments were performed with the robot's pushing motion with a variety of objects to generate sliding, falling over, bouncing, and rolling motions. For objects with single motion possibility, the robot tended to generate motions that induce the object motion. For objects with two motion possibilities, the robot evenly generated motions that induce the two object motions.

This paper proposes a model that enables a robot to predict and imitate the motions of another by reusing its body forward-inverse model. Our model includes three approaches: (i) projection of a self-forward model for predicting phenomena in the external environment (other individuals), (ii) ldquotriadic relationrdquo that is mediation by a physical object between self and others, (iii) introduction of infant imitation by a parent. The recurrent neural network with parametric bias (RNNPB) model is used as the robot's self forward-inverse model. A group of hierarchical neural networks are attached to the RNNPB model as ldquoconversion modulesrdquo. Experiments demonstrated that a robot with our model could imitate a human's motions by translating the viewpoint. It could also discriminate known/unknown motions appropriately, and associate whole motion dynamics from only one motion snap image.

Recent research in cognitive systems aims to uncover important aspects of biological cognitive processes and additionally formulate design principles for implementing artificially intelligent systems. Despite the increasing amount of research efforts addressing cognitive phenomena, the issue of time perception and how it is linked to other cognitive processes remains largely unexplored. In the current paper, we make a first attempt for studying artificial time perception by means of simulated robotic experiments. Specifically, we investigate a behavioral rule switching task consisting of repeating trials with dynamic temporal duration. An evolutionary process is used to search for neuronal mechanisms accomplishing the rule switching task taking also into account its particular temporal characteristics. Our repeated simulation experiments showed that (i) time perception and ordinary cognitive processes may co-exist in the system sharing the same neural resources, and (ii) time perception dynamics bias the functionality of neural mechanisms with other cognitive responsibilities. Finally, in the current paper we make contact of the obtained results with previous brain imaging studies on time perception, and we make predictions for possible time-related dynamics in the real brain.

We propose a reinforcement learning system designed to learn multiple different continuous state-action-space tasks. The system has been tested on a family of space-searching task akin to Morris water maze, but with obstacles. While exploring a task, the agent builds its internal model of the environment and approximates a state value function. For learning multiple tasks, we use a parametric bias switching mechanism in which the value of the parametric bias layer identifies the task for the agent. Each task has a specific parametric bias vector, and during training the vectors self-organize to reflect the structure of relationships between tasks in the task set. This mapping of the task set to parametric bias space can later be used to generate novel behaviors of the agent.

Tool-body assimilation is one of the intelligent human abilities. Through trial and experience, humans are capable of using tools as if they are part of their own bodies. This paper presents a method to apply a robot's active sensing experience for creating the tool-body assimilation model. The model is composed of a feature extraction module, dynamics learning module, and a tool recognition module. Self-Organizing Map (SOM) is used for the feature extraction module to extract object features from raw images. Multiple Time-scales Recurrent Neural Network (MTRNN) is used as the dynamics learning module. Parametric Bias (PB) nodes are attached to the weights of MTRNN as second-order network to modulate the behavior of MTRNN based on the tool. The generalization capability of neural networks provide the model the ability to deal with unknown tools. Experiments are performed with HRP-2 using no tool, I-shaped, T-shaped, and L-shaped tools. The distribution of PB values have shown that the model has learned that the robot's dynamic properties change when holding a tool. The results of the experiment show that the tool-body assimilation model is capable of applying to unknown objects to generate goal-oriented motions.

Consistency of object dynamics, which is related to reliable predictability, is an important factor for generating object manipulation motions. This paper proposes a technique to generate autonomous motions based on consistency of object dynamics. The technique resolves two issues: construction of an object dynamics prediction model and evaluation of consistency. The authors utilize Recurrent Neural Network with Parametric Bias to self-organize the dynamics, and link static images to the self-organized dynamics using a hierarchical neural network to deal with the first issue. For evaluation of consistency, the authors have set an evaluation function based on object dynamics relative to robot motor dynamics. Experiments have shown that the method is capable of predicting 90% of unknown object dynamics. Motion generation experiments have proved that the technique is capable of generating autonomous pushing motions that generate consistent rolling motions.

This paper presents a method to autonomously extract object features that describe their dynamics from active sensing experiences. The model is composed of a dynamics learning module and a feature extraction module. Recurrent Neural Network with Parametric Bias (RNNPB) is utilized for the dynamics learning module, learning and self-organizing the sequences of robot and object motions. A hierarchical neural network is linked to the input of RNNPB as the feature extraction module for extracting object features that describe the object motions. The two modules are simultaneously trained using image and motion sequences acquired from the robotpsilas active sensing with objects. Experiments are performed with the robotpsilas pushing motion with a variety of objects to generate sliding, falling over, bouncing, and rolling motions. The results have shown that the model is capable of extracting features that distinguish the characteristics of object dynamics.

It is generally thought that skilled behavior in human beings results from a functional hierarchy of the motor control system, within which reusable motor primitives are flexibly integrated into various sensori-motor sequence patterns. The underlying neural mechanisms governing the way in which continuous sensori-motor flows are segmented into primitives and the way in which series of primitives are integrated into various behavior sequences have, however, not yet been clarified. In earlier studies, this functional hierarchy has been realized through the use of explicit hierarchical structure, with local modules representing motor primitives in the lower level and a higher module representing sequences of primitives switched via additional mechanisms such as gate-selecting. When sequences contain similarities and overlap, however, a conflict arises in such earlier models between generalization and segmentation, induced by this separated modular structure. To address this issue, we propose a different type of neural network model. The current model neither makes use of separate local modules to represent primitives nor introduces explicit hierarchical structure. Rather than forcing architectural hierarchy onto the system, functional hierarchy emerges through a form of self-organization that is based on two distinct types of neurons, each with different time properties ("multiple timescales"). Through the introduction of multiple timescales, continuous sequences of behavior are segmented into reusable primitives, and the primitives, in turn, are flexibly integrated into novel sequences. In experiments, the proposed network model, coordinating the physical body of a humanoid robot through high-dimensional sensori-motor control, also successfully situated itself within a physical environment. Our results suggest that it is not only the spatial connections between neurons but also the timescales of neural activity that act as important mechanisms leading to functional hierarchy in neural systems.

This paper examines characteristics of interactive learning between human tutors and a robot having a dynamic neural-network model, which is inspired by human parietal cortex functions. A humanoid robot, with a recurrent neural network that has a hierarchical structure, learns to manipulate objects. Robots learn tasks in repeated self-trials with the assistance of human interaction, which provides physical guidance until the tasks are mastered and learning is consolidated within the neural networks. Experimental results and the analyses showed the following: 1) codevelopmental shaping of task behaviors stems from interactions between the robot and a tutor; 2) dynamic structures for articulating and sequencing of behavior primitives are self-organized in the hierarchically organized network; and 3) such structures can afford both generalization and context dependency in generating skilled behaviors.

Prediction of dynamic features is an important task for determining the manipulation strategies of an object. This paper presents a technique for predicting dynamics of objects relative to the robot's motion from visual images. During the learning phase, the authors use recurrent neural network with parametric bias (RNNPB) to self-organize the dynamics of objects manipulated by the robot into the PB space. The acquired PB values, static images of objects, and robot motor values are input into a hierarchical neural network to link the static images to dynamic features (PB values). The neural network extracts prominent features that induce each object dynamics. For prediction of the motion sequence of an unknown object, the static image of the object and robot motor value are input into the neural network to calculate the PB values. By inputting the PB values into the closed loop RNNPB, the predicted movements of the object relative to the robot motion are calculated sequentially. Experiments were conducted with the humanoid robot Robovie-IIs pushing objects at different heights. Reducted grayscale images and shoulder pitch angles were input into the neural network to predict the dynamics of target objects. The results of the experiment proved that the technique is efficient for predicting the dynamics of the objects.

We describe a means of human robot interaction based not on natural language but on "quasi symbols," which represent sensory-motor dynamics in the task and/or environment. It thus overcomes a key problem of using natural language for human-robot interaction - the need to understand the dynamic context. The quasi-symbols used are motion primitives corresponding to the attractor dynamics of the sensory-motor flow. These primitives are extracted from the observed data using the recurrent neural network with parametric bias (RNNPB) model. Binary representations based on the model parameters were implemented as quasi symbols in a humanoid robot, Robovie. The experiment task was robot-arm operation on a table. The quasi-symbols acquired by learning enabled the robot to perform novel motions. A person was able to control the arm through speech interaction using these quasi-symbols. These quasi symbols formed a hierarchical structure corresponding to the number of nodes in the model. The meaning of some of the quasi-symbols depended on the context, indicating that they are useful for human-robot interaction.

We present a connectionist model that combines motions and language based on the behavioral experiences of a real robot. Two models of recurrent neural network with parametric bias (RNNPB) were trained using motion sequences and linguistic sequences. These sequences were combined using their respective parameters so that the robot could handle many-to-many relationships between motion sequences and linguistic sequences. Motion sequences were articulated into some primitives corresponding to given linguistic sequences using the prediction error of the RNNPB model. The experimental task in which a humanoid robot moved its arm on a table demonstrated that the robot could generate a motion sequence corresponding to given linguistic sequence even if the motions or sequences were not included in the training data, and vice versa.

This paper proposes a novel model which enables a humanoid robot infant to discover other individual (e.g. human parent). In this work, the authors define “other individual” as an actor which can be predicted by a self-model. For modeling the developmental process of discovering ability, the following three approaches are employed. (i) Projection of a selfmodel for predicting other individual’s actions. (ii) Mediation by a physical object between self and other individual. (iii) Introduction of infant imitation by parent. For creating the self-model of a robot, we apply Recurrent Neural Network with Parametric Bias (RNNPB) model which can learn the robot’s body dynamics. For the other-model of a human, conventional hierarchical neural networks are attached to the RNNPB model as “conversion modules”. Our target task is a moving an object. For evaluation of our model, human discovery experiments by the robot projecting its self-model were conducted. The results demonstrated that our method enabled the robot to predict the human’s motions, and to estimate the human’s position fairly accurately, which proved its adequacy.

Robot imitation is a useful and promising alternative to robot programming. Robot imitation involves two crucial issues. The first is how a robot can imitate a human whose physical structure and properties differ greatly from its own. The second is how the robot can generate various motions from finite programmable patterns (generalization). This paper describes a novel approach to robot imitation based on its own physical experiences. Let us consider a target task of moving an object on a table. For imitation, we focused on an active sensing process in which the robot acquires the relation between the object's motion and its own arm motion. For generalization, we applied a recurrent neural network with parametric bias (RNNPB) model to enable recognition/generation of imitation motions. The robot associates the arm motion which reproduces the observed object's motion presented by a human operator. Experimental results demonstrated that our method enabled the robot to imitate not only motion it has experienced but also unknown motion, which proved its capability for generalization

Dynamic features play an important role in recognizing objects that have similar static features in colors and or shapes. This paper focuses on active sensing that exploits dynamic feature of an object. An extended version of the robot, Robovie-IIs, moves an object by its arm to obtain its dynamic features. Its issue is how to extract symbols from various kinds of temporal states of the object. We use the recurrent neural network with parametric bias (RNNPB) that generates self-organized nodes in the parametric bias space. The RNNPB with 42 neurons was trained with the data of sounds, trajectories, and tactile sensors generated while the robot was moving/hitting an object with its own arm. The clusters of 20 kinds of objects were successfully self-organized. The experiments with unknown (not trained) objects demonstrated that our method configured them in the PB space appropriately, which proves its generalization capability.

This study presents experiments on the imitative interactions between a small humanoid robot and a user. A mirror neuron model was implemented in a humanoid robot, based on the recurrent neural network model with parametric bias (RNNPB). The experiments showed that after the robot learns multiple cyclic movement patterns as embedded in the RNNPB, it can regenerate each pattern synchronously with the movements of a human who is demonstrating the corresponding movement pattern in front of the robot. On the other hand, the robot exhibits diverse interactive responses when the user demonstrates novel cyclic movement patterns. Those responses were analyzed and categorized. We propose that both the dynamics of coherence and incoherence between the robot and the user movements could enhance close interactions between them.

In this paper, we experimentally investigated the open-end interaction generated by the mutual adaptation between humans and robot. Its essential characteristic, incremental learning, is examined using the dynamical systems approach. Our research concentrated on the navigation system of a specially developed humanoid robot called Robovie and seven human subjects whose eyes were covered, making them dependent on the robot for directions. We used the usual feed-forward neural network (FFNN) without recursive connections and the recurrent neural network (RNN) for the robot control. Although the performances obtained with both the RNN and the FFNN improved in the early stages of learning, as the subject changed the operation by learning on its own, all performances gradually became unstable and failed. Next, we used a 'consolidation-learning algorithm' as a model of the hippocampus in the brain. In this method, the RNN was trained by both new data and the rehearsal outputs of the RNN not to damage the contents of current memory. The proposed method enabled the robot to improve performance even when learning continued for a long time (open-end). The dynamical systems analysis of RNNs supports these differences and also showed that the collaboration scheme was developed dynamically along with succeeding phase transitions.

A novel approach to human-robot collaboration based on quasi-symbolic expressions is proposed. The target task is navigation in which a person with his or her covered and a humanoid robot collaborate in a context-dependent manner. The robot uses a recurrent neural net with parametric bias (RNNPB) model to acquire the behavioral primitives, which are sensory-motor units, composing the whole task. The robot expresses the PB dynamics as primitives using symbolic sounds, and the person influences these dynamics through tactile sensors attached to the robot. Experiments with six participants demonstrated that the level of influence the person has on the PB dynamics is strongly related to task performance, the person's subjective impressions, and the prediction error of the RNNPB model (task stability). Simulation experiments demonstrated that the subjective impressions of the correspondence between the utterance sounds (the PB values) and the motions were well reproduced by the rehearsal of the RNNPB model.

In this paper, we investigated interactive learning between human subjects and robot experimentally, and its essential characteristics are examined using the dynamical systems approach. Our research concentrated on the navigation system of a specially developed humanoid robot called Robovie and seven human subjects whose eyes were covered, making them dependent on the robot for directions. We compared the usual feed-forward neural network (FFNN) without recursive connections and the recurrent neural network (RNN). Although the performances obtained with both the RNN and the FFNN improved in the early stages of learning, as the subject changed the operation by learning on its own, all performances gradually became unstable and failed. Results of a questionnaire given to the subjects confirmed that the FFNN gives better mental impressions, especially from the aspect of operability. When the robot used a consolidation-learning algorithm using the rehearsal outputs of the RNN, the performance improved even when interactive learning continued for a long time. The questionnaire results then also confirmed that the subject's mental impressions of the RNN improved significantly. The dynamical systems analysis of RNNs supports these differences.

We present a novel connectionist model for acquiring the semantics of a simple language through the behavioral experiences of a real robot. We focus on the “compositionality” of semantics, a fundamental character- istic of human language, which is the ability to understand the meaning of a sentence as a combination of the meanings of words. We also pay much attention to the “embodiment” of a robot, which means that the robot should acquire semantics which matches its body, or sensory-motor system. The essential claim is that an embodied compositional semantic representation can be self-organized from generalized correspondences between sentences and behavioral patterns. This claim is examined and confirmed through simple experiments in which a robot generates corre- sponding behaviors from unlearned sentences by analogy with the corre- spondences between learned sentences and behaviors.

This paper investigates how behavior primitives are self-organized in a neural network model utilizing a distributed representation scheme. The model is characterized by so-called parametric biases which adaptively modulate the encoding of different behavior patterns in a single recurrent neural net (RNN). Our experiments, using a real robot arm, showed that a set of end-point and oscillatory behavior patterns are learned by self-organizing fixed points and limit cycle dynamics that form behavior primitives. It was also found that diverse novel behavior patterns can be generated by modulating the parametric biases arbitrarily. Our analysis showed that such diversity in behavior generation emerges because a nonlinear map is self-organized between the space of parametric biases and that of the behavior patterns. The origin of the observed nonlinearity from the distributed representation is discussed. This paper investigates how behavior primitives are self-organized in a neural network model utilizing a distributed representation scheme. Our robot experiments showed that a set of end-point and oscillatory behavior patterns are learned by self-organizing fixed points and limit cycle dynamics that form behavior primitives. It was also found that diverse novel behavior patterns, in addition to previously learned patterns, can be generated by taking advantage of nonlinear effects that emerge from the distributed representation.

We propose that the ability to extract regularities from time series through prediction learning can be enhanced if we use a hierarchical architecture in which higher layers are trained to predict the internal state of lower layers when such states change significantly. This hierarchical organization has two functions: (a) it forces the system to recode sensory information progressively so as to enhance useful regularities and filter out useless information; and (b) it progressively reduces the length of the sequences which should be predicted going from lower to higher layers. This, in turn, allows higher levels to extract higher-level regularities which are hidden at the sensory level. By training an architecture of this type to predict the next sensory state of a robot navigating in an environment divided into two rooms, we show how the first-level prediction layer extracts low-level regularities such as 'walls', 'corners' and 'corridors', while the second-level prediction layer extracts higher-level regularities such as 'the left side wall of the large room'. The extraction of these regularities allows the robot to localize its position in the environment and to detect changes in the environment (e.g. the presence of a new object or the fact that a door has been closed).

This paper discusses how a behavior-based robot can construct a "symbolic process" that accounts for its deliberative thinking processes using models of the environment. The paper focuses on two essential problems; one is the symbol grounding problem and the other is how the internal symbolic processes can be situated with respect to the behavioral contexts. We investigate these problems by applying a dynamical system's approach to the robot navigation learning problem. Our formulation, based on a forward modeling scheme using recurrent neural learning, shows that the robot is capable of learning grammatical structure hidden in the geometry of the workspace from the local sensory inputs through its navigational experiences. Furthermore, the robot is capable of generating diverse action plans to reach an arbitrary goal using the acquired forward model which incorporates chaotic dynamics. The essential claim is that the internal symbolic process, being embedded in the attractor, is grounded since it is self-organized solely through interaction with the physical world. It is also shown that structural stability arises in the interaction between the neural dynamics and the environmental dynamics, which accounts for the situatedness of the internal symbolic process, The experimental results using a mobile robot, equipped with a local sensor consisting of a laser range finder, verify our claims.

This study shows a novel approach for learning sensory- and task-based navigation of a mobile robot. The learning of task-based behavior is formulated as an embedding problem of dynamical systems: the desired trajectories in a task space should be embedded into an adequate sensory-based internal state space so that unique mapping from the internal state space to the motor command can be established. We discuss how such internal state space and its mapping can be self-organized by means of neural learning methodologies. Physical experiment by a mobile robot with laser range sensor was conducted, which showed that learning of homing and cyclic routing tasks were successfully realized on two different neural network (NN) architectures; the time delayed type and the recurrent type. The navigation was inherently robust against miscellaneous noise since those tasks were embedded in the global attractor dynamics of a fixed point and limit cycling.>