In this study, we propose a shared control method for teleoperated mobile robots using brain-machine interfaces (BMI). The control commands generated through BMI for robot operation face issues of low input frequency, discreteness, and uncertainty due to noise. To address these challenges, our method estimates the user’s intended goal from their commands and uses this goal to generate auxiliary commands through the autonomous system that are both at a higher input frequency and more continuous. Furthermore, by defining the confidence level of the estimation, we adaptively calculated the weights for combining user and autonomous commands, thus achieving shared control. We conducted navigation experiments in both simulated environments and participant experiments in real environments including user ratings, using a pseudo-BMI setup. As a result, the proposed method significantly reduced obstacle collisions in all experiments. It markedly shortened path lengths under almost all conditions in simulations and, in participant experiments, especially when user inputs become more discrete and noisy (p<0.01). Furthermore, under such challenging conditions, it was demonstrated that users could operate more easily, with greater confidence, and at a comfortable pace through this system.

When individuals communicate, they use different vocabularies, speaking speeds, facial expressions, and gestural languages, depending on those with whom they are speaking. This study focuses on the age of the speaker as a factor that affects the style of communication. We collected a multimodal dialogue corpus with various speaker ages. We used travel as the topic, as it interests people of all ages, and we set up a task based on a tourism consultation between an operator and a customer at a travel agency. This article presents the details of the dialogue task, collection procedures and annotations, and analysis of the characteristics of the dialogues and facial expressions, focusing on the age of the speakers. The results of the analysis suggest that the adult speakers have more independent opinions, the older speakers express their opinions more frequently than other age groups, and those in the operator role smile more frequently at minors.

This paper discusses the tele-operation system for multiple mobile manipulators. If a single person could freely tele-operate multiple mobile manipulators simultaneously, it would be a great step toward the goal of “avatar-symbiotic society” allowing people to live beyond the constraints of their bodies, space, and time. At present, however, such a tele-operation system has not been developed. Therefore, we built a prototype system to tele-operate two mobile manipulators and conducted a subject experiment on a pick-and-place task to investigate the tele-operator's task performance, workload and gaze behavior. By analyzing these results, we obtained guidelines to design tele-operation system for multiple robots.

When individuals communicate with each other, they use different vocabulary, speaking speed, facial expressions, and body language depending on the people they talk to. This paper focuses on the speaker’s age as a factor that affects the change in communication. We collected a multimodal dialogue corpus with a wide range of speaker ages. As a dialogue task, we focus on travel, which interests people of all ages, and we set up a task based on a tourism consultation between an operator and a customer at a travel agency. This paper provides details of the dialogue task, the collection procedure and annotations, and the analysis on the characteristics of the dialogues and facial expressions focusing on the age of the speakers. Results of the analysis suggest that the adult speakers have more independent opinions, the older speakers more frequently express their opinions frequently compared with other age groups, and the operators expressed a smile more frequently to the minor speakers.

Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information. In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.

Personality estimation of others is a critical ability to communicate with each other. It enables robots to interact with humans and provides the former the ability to predict the intentions of the latter. Many researchers have developed personality estimation mechanisms. However, the estimation method for toddlers' personality, such as the dominance of their innate temperament, has not been proposed yet. In this paper, we proposed an estimation model of toddlers' temperament based on interaction data with a teleoperated childcare robot, ChiCaRo. The proposed method utilized the feature selection algorithm to increase estimation accuracy. Additionally, we employed an explainable AI model called Shapley additive explanations (SHAP) to understand which features from the interaction were important in terms of temperament estimation. The proposed estimation model demonstrated over 85% estimation accuracy for the average of all temperament factors. The experimental results of SHAP provided an understandable relation between the features and temperament factors and indicated that similar feature values from interaction videos used in child personality estimation could also be used for the temperament estimation of toddlers.

The purpose of this study is for a robot to learn picking motions in a logistics warehouse environment. The picking operation performed by a robot often fails owing to the inclination of items placed on a shelf, as well as the minimum clearance between the products and their vinyl packaging. Therefore, we considered acquiring a specific motion trajectory by reinforcement learning. However, because numerous types of items are handled in logistics warehouses, efficient learning is required. Therefore, in this research, we propose a method to efficiently exploration for learning picking an object by determining a focus exploration area for learning based on previous results of different objects.

Humans perceive continuous high-dimensional information by dividing it into significant segments such as words and units of motion. We believe that such unsupervised segmentation is also important for robots to learn topics such as language and motion. To this end, we previously proposed a hierarchical Dirichlet process-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). However, an important drawback to this model is that it cannot divide high-dimensional time-series data. Further, low-dimensional features must be extracted in advance. Segmentation largely depends on the design of features, and it is difficult to design effective features, especially in the case of high-dimensional data. To overcome this problem, this paper proposes a hierarchical Dirichlet process-variational autoencoder-Gaussian process-hidden semi-Markov model (HVGH). The parameters of the proposed HVGH are estimated through a mutual learning loop of the variational autoencoder and our previously proposed HDP-GP-HSMM. Hence, HVGH can extract features from high-dimensional time-series data, while simultaneously dividing it into segments in an unsupervised manner. In an experiment, we used various motion-capture data to show that our proposed model estimates the correct number of classes and more accurate segments than baseline methods. Moreover, we show that the proposed method can learn latent space suitable for segmentation.

In this paper, we propose a method for motion learning aimed at the execution of autonomous household chores by service robots in real environments. For robots to act autonomously in a real environment, it is necessary to define the appropriate actions for the environment. However, it is difficult to define these actions manually. Therefore, body motions that are common to multiple actions are defined as motion primitives. Complex actions can then be learned by combining these motion primitives. For learning motion primitives, we propose a reference-point and object-dependent Gaussian process hidden semi-Markov model (RPOD-GP-HSMM). For verification, a robot is teleoperated to perform the actions included in several domestic household chores. The robot then learns the associated motion primitives from the robot's body information and object information.

Humans recognize perceived continuous information by dividing it into significant segments such as words and unit motions. We believe that such unsupervised segmentation is also an important ability that robots need to learn topics such as language and motions. Hence, in this paper, we propose a method for dividing continuous time-series data into segments in an unsupervised manner. To this end, we proposed a method based on a hidden semi-Markov model with Gaussian process (GP-HSMM). If Gaussian processes, which are nonparametric models, are used, unit motion patterns can be extracted from complicated continuous motion. However, this approach requires the number of classes of segments in the time-series data in advance. To overcome this problem, in this paper, we extend GP-HSMM to a nonparametric Bayesian model by introducing a hierarchical Dirichlet process (HDP) and propose the hierarchical Dirichlet processes-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). In the nonparametric Bayesian model, an infinite number of classes is assumed and it becomes difficult to estimate the parameters naively. Instead, the parameters of the proposed HDP-GP-HSMM are estimated by applying slice sampling. In the experiments, we use various synthetic and motion-capture data to show that our proposed model can estimate a more correct number of classes and achieve more accurate segmentation than baseline methods.

Having emotions is essential for robots in order for them to understand and sympathize with people's feelings. In addition, it may allow robots to be accepted in human society. The role of emotions in decision-making is another important perspective. In this paper, a model of emotions is proposed based on various neurological and psychological findings related to empathic communication between humans and robots. Subsequently, a decision-making mechanism based on affects using convolutional long short-term memory and deep deterministic policy gradient is examined. We set a “facial expression” task simulating mother-child interactions and verified emotion differentiation during the task.

We normally tend to think that we, as human beings, act as if we have intention to achieve a goal. However, in the situation of cooperating with other people, it is difficult to come up with a mechanism for predicting and adjusting each other's intention when we think that the goal is given top-down. Thinking that intention comes from context may help to solve that problem. In this study, we attempt to model an action learning mechanism that generates/predicts self-intention using deep Q-network and recurrent neural network. And then we show such a mechanism makes it possible for the agent to predict intention of others and goals.

Humans interact with one another daily and learn from their experiences via observation and interaction. To create robots that can coexist with humans, it is important that they learn how to appropriately interact in the human community. In this paper, we propose a novel model, the coupled Gaussian process hidden semi-Markov model (GP-HSMM), which enables robots to learn rules of interaction between two persons by observing them in an unsupervised manner. The continuous motions of the persons are segmented into discrete actions based on GP-HSMM, and the relationships between the actions are extracted. Moreover, all corresponding actions are not simultaneously conducted by two persons during actual interaction. Thus, the coupled GP-HSMM accounts for these lags. We conduct experiments using the motion data of interaction games. Experimental results showed that the coupled GP-HSMM can estimate actions, lags between them and their relationships.

For presentation purposes, gestures play an exceptionally important role in improving the information transmission effect. It has been demonstrated that the body language expressing the enthusiasm and intention of the presenter affects the success of the presentation and the impression on the audience. For these reasons, presentation robots are required to perform such movements; however, manual design of these movements is a difficult task. In this research, we propose a method to model the relationship between speech prosodic information and motion using a recurrent neural network, and directly generate appropriate motions using the prosodic information. This study also proposes a method for generating motions that convey the meaning of specific words. We implement the proposed method on the “Pepper” robot to evaluate its performance.

Human intelligence is deeply dependent on its physical body, and its development requires interaction between its own body and surrounding environment including other agents. However, it is still an open problem that how we can integrate the low level motor control and the high level symbol manipulation system. One of our research goals in the area called "symbol emergence in robotics" is to build a computational model of human intelligence from the motor control to the high level symbol manipulation. In this talk, an unsupervised on-line learning algorithm, which uses a hierarchical Bayesian framework for categorizing multimodal sensory signals such as audio, visual, and haptic information by robots, is introduced at first. The robot uses its physical body to grasp and observe an object from various viewpoints as well as listen to the sound during the observation. The basic algorithm for intelligence is to categorize the collected multimodal data so that the robot can infer unobserved information better and we call the generated categorizes as multimodal concepts. The latter half of this talk discusses an integrated computational model of human intelligence from the motor control to the high level cognition. The core idea is to integrate the multimodal concepts and reinforcement learning. Furthermore, this talk attempts to model communication within the same framework since the self-other discrimination process can be seen as the multimodal categorization of sensory-motor signals.

This paper proposes a new approach for research on chat-like conversational systems that enable robots to acquire physically grounded knowledge through natural interaction with humans. The proposed approach combines research on chat-like conversational systems, language acquisition, and symbol grounding in order to realize physically situated and natural human-robot interaction. In contrast to previous approaches for chat-like conversation, the proposed approach focuses on utterances which are situated in physical environments surrounding humans and robots. Based on the proposed approach, we develop a concrete method that enables robots to learn object image concepts and the words describe them from object-teaching utterances made by humans. The method is composed of two processes:(1) the detection of object-teaching utterances from chat-like conversation and (2) the learning of object image concepts and the words describing them. It applies a linear support vector machine, multimodal hierarchical Dirichlet process, and term frequency-inverse document frequency process. The experimental results show that the method enabled robots to learn object image concepts and the words that describe them through multimodal chat-like interactions with humans.

We propose a technique to estimate a child's extraversion and agreeableness for social robots that interact with children. The proposed approach observed children's behavior using only the robot's sensors, without any sensor networks in the environment. An RGBD sensor was used to track and identify children's facial expressions. Children's interactions with the robot were observed, such as their distance from the robot and the duration of their eye contact, because such information would provide clues to estimate their personality. Data were collected when a robot, tele-operated by preschool teachers, interacted with kindergarten children individually. The data from 29 children was used to successfully estimate the children's personality compared to chance rates.

The facial expression recognition has been a topic of active researches given a result the proposal of several efficient algorithms; however, in most cases they remain limited to controlled conditions situations. In this study, we tackle the challenge of recognizing emotions through the facial expression into activities in-the-wild adding the accuracy rate for each expression. To this end we an algorithm that allows accurate face expression recognition in an uncontrolled environment, that means different kind of illumination, backgrounds, occlusions, face's profiles, etc. Proposed system firstly detects different profile of face (left, frontal and right), Then it uses only the frames in which the face profile is frontal, in the next step the face regions of interest (ROI) are segmented automatically to carry out the feature extraction. We use a classifier based on clustering, it has the advantage that if a new class (emotion) is added, it is not necessary to train this completely. Proposed system was evaluated using short video clips of several pictures together with description sentences describing the main activity in the video. The evaluation results show that the proposed scheme is able to recognize the face's profiles with the recognition rate to approximately 93% and principal emotions in unconstrained video sequences.

Social robots are used for interacting with children. Their novelty often stimulates the interest of children and encourages such interaction as playing with them. However, some children do not interact with such robots and instead strongly reject interaction with them. Understanding such rejection behaviors of children has important design implications for social robots that are supposed to interact with children. For this purpose, we investigated what kinds of rejection behaviors appeared in different kinds of robots in a play room environment.

This paper reports a tele-presence robot named ChiCaRo, which is designed for interaction with babies and toddlers. ChiCaRo can physically interact with babies and toddlers by moving around and using its small hand. We conducted a field trial at a playroom where babies and toddlers can freely play to investigate ChiCaRo's effectiveness. In the experiment adult participants interacted with their babies and toddlers by ChiCaRo and another robot. The adult participants evaluated ChiCaRo highly in the context of remote interaction with their babies and toddlers.

Recent studies have shown that human beings unconsciously use signals that represent their thoughts and/or intentions when communicating with each other. These signals are known as “honest signals.” This study involves the use of a sociometer to capture multimodal data resulting from the interaction between humans. These data are then used to model the interaction using a multimodal hierarchical Dirichlet process hidden Markov model, which is then implemented in the robot. The model enables robots to generate “honest signals” and to interact in a natural manner.

One of the biggest challenges in intelligent robotics is to build robots that can learn to use language. To this end, we think that the practical long-term on-line concept/word learning algorithm for robots is a key issue to be addressed. In this paper, we develop an unsupervised on-line learning algorithm that uses Bayesian nonparametrics for categorizing multimodal sensory signals such as audio, visual, and haptic information for robots. The robot uses its physical body to grasp and observe an object from various viewpoints as well as listen to the sound during the observation. The most important property of the proposed framework is to learn multimodal concepts and the language model simultaneously. This mutual learning framework of concepts and language significantly improves both speech recognition and multimodal categorization performances. We conducted a long-term experiment where a human subject interacted with a real robot over 100 hours using 499 objects. Some interesting results of the experiment are discussed in this paper.

Muhammad Attamimi, Yuji Ando, Tomoaki Nakamura, Takayuki Nagai, Daichi Mochihashi, Ichiro Kobayashi, Hideki Asoh. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015.

We propose a method for a robot to form various concepts. The robot uses its embodiment to obtain visual, auditory, and haptic information by grasping, shaking, and observing objects. At the same time, a user teaches the robot object features through speech. From these kinds of information, the robot can form object concepts. The information obtained by the robot is converted into Bag-of-Words representations, which are classified into categories. The proposed method is based on a stochastic model, and objects can be classified by estimating their parameters. We introduce the Chinese restaurant process into the multimodal hierarchical Dirichlet process. This model is an infinite mixture of models and enables the robot to form various concepts such as object type, color, and so on. Therefore, the robot can form not only object concepts but also various concepts that are not represented by object concepts (e.g., color). Also, because the proposed method is based on a stochastic model, it makes it possible for the robot to estimate category and unobserved information of unseen objects. We implement the proposed method on a robot and show that it can form various concepts and perform various estimations for unseen objects.

Intimacy is a very important factor not only for the communication between humans but also for the communication between humans and robots. In this research we propose an action decision model based on others' friendliness value, which can be estimated using the mental models of others. We examine the mutual adaptation process of two agents, each of which has its own model of others, through the interaction between them.

Though communication is one of our basic activity, it is not always that we can interact effectively. It is well known that a key point for a successful interaction is the inclusion of other with a good mood. It means acquisition of other's interest is a precondition for a successful communication.

It is difficult to design robotic playmates for introverted children. Therefore, we examined how a robot should play with such shy children. In this study, we hypothesized and tested an effective play strategy for building a good relationship with shy children. We conducted an experiment with 5- to 6-year-old children and a humanoid robot teleoperated by a preschool teacher. We developed a valid play strategy for shy children.

Similar to relationships between humans, a person desiring to form a good relationship with a robot needs to be able to empathize with it. However, the specific kinds of human-robot interactions that would arouse and enhance empathy for the robot in the user's mind have not yet been clarified. In addition, the human behavioral traits that may be regarded as indices of empathy have not been investigated extensively. In an attempt to address these two issues, a preliminary experiment on empathy in human-robot interaction is conducted. The results suggest that the actions of naming or comforting a robot could contribute to enhancing its user's empathy and that eye fixation could be used as an index of empathy even when the use of a subjective index is inconclusive.

This work proposes a new approach to realize embodied and multimodal HRI between virtual robot and real world human for HRI challenges in RoboCup @Home.

In the field of intelligent robotics, object handling by robots can be achieved by capturing not only the object concept through object categorization, but also other concepts (e.g., the movement while using the object), as well as the relationship between concepts. Moreover, capturing the concepts of places and people is also necessary to enable the robot to gain real-world understanding. In this study, we propose multi-layered multimodal latent Dirichlet allocation (mMLDA) to realize the formation of various concepts, and the integration of those concepts, by robots. Because concept formation and integration can be conducted by mMLDA, the formation of each concept affects others, resulting in a more appropriate formation. Another issue to be addressed in this paper is the language acquisition by the robots. We propose a method to infer which words are originally connected to a concept using mutual information between words and concepts. Moreover, the order of concepts in teaching utterances can be learned using a simple Markov model, which corresponds to grammar. This grammar can be used to generate sentences that represent the observed information. We report the results of experiments to evaluate the effectiveness of the proposed method.

The influence of holding hands on the relationship building process between children and robots is investigated in this study. In particular, at the first meeting, it is difficult for a child to be open if he/she starts to rebuff the robot partner. This significantly degrades the possibility of the child forming a friendship relationship with the robot. Thus, the initial approach of the robot to the child in the early stage of the relationship building process should be appropriate. We hypothesize that physical embodied communication, such as walking hand in hand, improves the relationship between children and robots. A holding hands system was implemented in a real robot, and an experiment was conducted at a kindergarten to validate our hypothesis. The results strongly support our hypothesis.

Humans develop their concept of an object by classifying it into a category, and acquire language by interacting with others at the same time. Thus, the meaning of a word can be learnt by connecting the recognized word and concept. We consider such an ability to be important in allowing robots to flexibly develop their knowledge of language and concepts. Accordingly, we propose a method that enables robots to acquire such knowledge. The object concept is formed by classifying multimodal information acquired from objects, and the language model is acquired from human speech describing object features. We propose a stochastic model of language and concepts, and knowledge is learnt by estimating the model parameters. The important point is that language and concepts are interdependent. There is a high probability that the same words will be uttered to objects in the same category. Similarly, objects to which the same words are uttered are highly likely to have the same features. Using this relation, the accuracy of both speech recognition and object classification can be improved by the proposed method. However, it is difficult to directly estimate the parameters of the proposed model, because there are many parameters that are required. Therefore, we approximate the proposed model, and estimate its parameters using a nested Pitman-Yor language model and multimodal latent Dirichlet allocation to acquire the language and concept, respectively.

The recording of lecture videos and disseminating them is becoming popular. However, recording and post-processing daily lectures still demands much human effort and financial resources. Our goal is to develop a cost-effective and labor-saving video processing system that can record lectures in high-definition and process them for additional analysis and publishing. Our main idea is that high-definition recording itself is labor-saving. A stationary high-definition camcorder can record a lecturer and an entire classroom clearly so that notes can be read on a whiteboard and projector screen. Therefore, no camera operator is necessary during recording. We succeeded in implementing a cost-effective system without expensive hardware components for capturing a high-definition video signal. In this contribution, we describe the design and operational results of our system, which has recorded about 20 lectures every school week since 2010.

In recent studies, it has been revealed that robots can form concepts and understand the meanings of words through inference. The key idea underlying these studies is “multimodal categorization” of a robot's experience. However, previous studies considered only nonhierarchical categorization methods, which led to nonhierarchical concept structures. Our concepts have a hierarchical structure, thus ensuring that the resulting inferences are more efficient and accurate. In this paper, we propose a novel hierarchical categorization method. The method involves extending multimodal latent Dirichlet allocation (MLDA) to hierarchical MLDA using the nested Chinese restaurant process, which makes it possible for robots to acquire concepts in a hierarchical structure. We show that a robot can form a hierarchical concept structure based on self-obtained multimodal information. Moreover, by focusing on the common features of each category in the hierarchy, the robot is able to infer unobserved information including word meanings.

One of the biggest challenges in intelligent robotics is to build robots that can understand and use language. Such robots will be a part of our everyday life; at the same time, they can be of great help to investigate the complex mechanism of language acquisition by infants in constructive approach. To this end, we think that the practical long-term on-line concept/word learning algorithm for robots and the interactive learning framework are the key issues to be addressed. In this paper we develop a practical on-line learning algorithm that solves three remaining problems in our previous study. We also propose an interactive learning framework, in which the proposed on-line learning algorithm is embedded. The main contribution of this paper is to develop such a practical learning framework, and we test it on a real robot platform to show its potential toward the ultimate goal.

The human understanding of things is based on prediction which is made through concepts formed by the categorization of experience. To mimic this mechanism in robots, multimodal categorization, which enables the robot to form concepts, has been studied. On the other hand, segmentation and categorization of human motions have also been studied to recognize and predict future motions. This paper addresses the issue on how these different kinds of concepts are integrated to generate higher level concepts and, more importantly, on how the higher level concepts affect the formation of each lower level concept. To this end, we propose the multi-layered multimodal latent Dirichlet allocation (mMLDA), which is an expansion of the MLDA to learn and represent the hierarchical structure of concepts. We also examine a simple integration model and compare it with the mMLDA. The experimental results reveal that the mMLDA leads to a better inference performance and, indeed, forms higher level concepts which integrate motions and objects that are necessary for real-world understanding.

In this study, we propose a method for concept formation and word acquisition for robots. The proposed method is based on multimodal latent Dirichlet allocation (MLDA) and the nested Pitman-Yor language model (NPYLM). A robot obtains haptic, visual, and auditory information by grasping, observing, and shaking an object. At the same time, a user teaches object features to the robot through speech, which is recognized using only acoustic models and transformed into phoneme sequences. As the robot is supposed to have no language model in advance, the recognized phoneme sequences include many phoneme recognition errors. Moreover, the recognized phoneme sequences with errors are segmented into words in an unsupervised manner; however, not all words are necessarily segmented correctly. The words including these errors have a negative effect on the learning of word meanings. To overcome this problem, we propose a method to improve unsupervised word segmentation and to reduce phoneme recognition errors by using multimodal object concepts. In the proposed method, object concepts are used to enhance the accuracy of word segmentation, reduce phoneme recognition errors, and correct words so as to improve the categorization accuracy. We experimentally demonstrate that the proposed method can improve the accuracy of word segmentation and reduce the phoneme recognition error and that the obtained words enhance the categorization accuracy.

We have been developing a typing training system and we tried to support touch-typing training, where eye-gaze must be detected for checking the trainee is not looking at the keyboard. The eye-gaze detecting function was designed to use a normal PC with a built-in or general USB camera and to discriminate at least whether the trainee is looking at the display or the keyboard in real-time. We developed simple and quick discriminating method consisting two stages. The first stage is detecting eye regions by template matching, and the second stage is discriminating eye-gaze by shape matching. After a short-time calibration for making the trainee's own template image and determining the threshold for shape matching, the eye-gaze is successfully detected more than 15 times per a second.

Web-based laboratories have become so popular in educational institutions for the great benefits these labs bring and ease of use. They have been successfully implemented in a wide range of subjects, but we still need to build training facilities for the uncovered/new branches of learning. In general, the structure of the laboratory varies depending on the subject area; however, in many cases, especially in the case of ICT subjects, it is possible to use the same configurations for more than one course-topic, Networking and Internet Security for example. This means that we only need to create a “prototype” then we can have as many labs as we wish by making “copies” of it. This is not possible with all laboratory-types, but certain technologies make it doable. In this paper, we describe how we used Virtualization Technology (normal and nested) to achieve this successfully. In our system the laboratory components are constructed using Virtual Machines, and then packaged to make one prototype from which we could create a number of labs for different ICT courses.

We propose a playmate robot system that can play with a child. Unlike many therapeutic service robots, our proposed playmate system is implemented as a functionality of the domestic service robot with a high degree of freedom. This implies that the robot can play high-level games with children, i.e., beyond therapeutic play, using its physical features. The proposed system currently consists of ten play modules, including a chatbot with eye contact, card playing, and drawing. The algorithms of these modules are briefly discussed in this paper. To sustain the player's interest in the system, we also propose an action-selection strategy based on a transition model of the child's mental state. The robot can estimate the child's state and select an appropriate action in the course of play. A portion of the proposed algorithms was implemented on a real robot platform, and experiments were carried out to design and evaluate the proposed system.

In this paper, we propose an online algorithm for multimodal categorization based on the autonomously acquired multimodal information and partial words given by human users. For multimodal concept formation, multimodal latent Dirichlet allocation (MLDA) using Gibbs sampling is extended to an online version. We introduce a particle filter, which significantly improve the performance of the online MLDA, to keep tracking good models among various models with different parameters. We also introduce an unsupervised word segmentation method based on hierarchical Pitman-Yor Language Model (HPYLM). Since the HPYLM requires no predefined lexicon, we can make the robot system that learns concepts and words in completely unsupervised manner. The proposed algorithms are implemented on a real robot and tested using real everyday objects to show the validity of the proposed system.

In this study, we propose a system to enable robots to navigate more efficiently to target objects in order to carry out mobile manipulation tasks. Because a robot arm has limited reach, a target object cannot be grasped if the distance to the object exceeds the reach. At the same time, navigation errors may increase as the robot moves around in an environment. To mitigate these problems, we propose the use of maps that express the arm reachability and navigation reachability. Places with minimum probability of navigation errors and maximum tolerable navigation errors for the reach of the arm can be determined by integrating these two maps. We also consider the fact that the target object position is not always known. To cope with this problem, we introduce an object existence map that represents the ambiguity of the target object position, considering the past position. If the ambiguity is large, it is possible to select a place from where the object searching task can be performed with a wide angle of view by the robot. Efficient object searching can be achieved by changing the task objective from searching to grasping and vice versa. We conducted some mobile manipulation tasks to evaluate the proposed method. The results showed that the proposed system can perform such tasks effectively.

The formation of categories, which constitutes the basis of developing concepts, requires multimodal information with a complex structure. We propose a model called the bag of multimodal hierarchical Dirichlet processes (BoMHDP), which enables robots to form a variety of multimodal categories. The BoMHDP model is a collection of a large number of MHDP models, each of which has a different set of weights for sensory information. The weights work to realize selective attention and enable the formation of various types of categories (e.g., object, haptic, and color). The BoMHDP model is an extension of the HDP, and categorization is unsupervised. However, categories that are not natural for humans are also formed. Therefore, only the significant categories are selected through interaction between the user and the robot. At the same time, words obtained during the interaction are connected to the categories. Finally, categories, which are represented by words, are selected. The BoMHDP model was implemented on a robot platform and a preliminary experiment was conducted to validate it. The results revealed that various categories can be formed with the BoMHDP model. We also analyzed the formed conceptual structure by using multidimensional scaling. The results indicate that the complex conceptual structure was represented reasonably well with the BoMHDP model.

In this paper a novel framework for multimodal categorization using Bag of multimodal LDA models is proposed. The main issue, which is tackled in this paper, is granularity of categories. The categories are not fixed but varied according to context. Selective attention is the key to model this granularity of categories. This fact motivates us to introduce various sets of weights to the perceptual information. Obviously, as the weights change, the categories vary. In the proposed model, various sets of weights and model structures are assumed. Then the multimodal LDA-based categorization is carried out many times that results in a variety of models. In order to make the categories (concepts) useful for inference, significant models should be selected. The selection process is carried out through the interaction between the robot and the user. These selected models enable the robot to infer unobserved properties of the object. For example, the robot can infer audio information only from its appearance. Furthermore, the robot can describe appearance of any objects using some suitable words, thanks to the connection between words and perceptual information. The proposed algorithm is implemented on a robot platform and preliminary experiment is carried out to validate the proposed algorithm.

This paper proposes a robot that acquires multi-modal information, i.e. auditory, visual, and haptic information, fully autonomous way using its embodiment. We also propose an online algorithm of multimodal categorization based on the acquired multimodal information and words, which are partially given by human users. The proposed framework makes it possible for the robot to learn object concepts naturally in everyday operation in conjunction with a small amount of linguistic information from human users. In order to obtain multimodal information, the robot detects an object on a fla surface. Then the robot grasps and shakes it for gaining haptic and auditory information. For obtaining visual information, the robot uses a hand held small observation table, so that the robot can control the viewpoints for observing the object. As for the multimodal concept formation, the multimodal LDA using Gibbs sampling is extended to the online version in this paper. The proposed algorithms are implemented on a real robot and tested using real everyday objects in order to show validity of the proposed system.

In this paper, we propose a nonparametric Bayesian framework for categorizing multimodal sensory signals such as audio, visual, and haptic information by robots. The robot uses its physical embodiment to grasp and observe an object from various viewpoints as well as listen to the sound during the observation. The multimodal information enables the robot to form human-like object categories that are bases of intelligence. The proposed method is an extension of Hierarchical Dirichlet Process (HDP), which is a kind of nonparametric Bayesian models, to multimodal HDP (MHDP). MHDP can estimate the number of categories, while the parametric model, e.g. LDA-based categorization, requires to specify the number in advance. As this is an unsupervised learning method, a human user does not need to give any correct labels to the robot and it can classify objects autonomously. At the same time the proposed method provides a probabilistic framework for inferring object properties from limited observations. Validity of the proposed method is shown through some experimental results.

This paper presents a method for learning novel objects from audio-visual input. Objects are learned using out-of-vocabulary word segmentation and object extraction. The latter half of this paper is devoted to evaluations. We propose the use of a task adopted from the RoboCup@Home league as a standard evaluation for real world applications. We have implemented proposed method on a real humanoid robot and evaluated it through a task called “Supermarket”. The results reveal that our integrated system works well in the real application. In fact, our robot outperformed the maximum score obtained in RoboCup@Home 2009 competitions.

This paper presents a novel 3D measurement system, which yields both depth and color information in real time, by calibrating a time-of-flight and two CCD cameras. The problem of occlusions is solved by the proposed fast occluded-pixel detection algorithm. Since the system uses two CCD cameras, missing color information of occluded pixels is covered by one another. We also propose a robust object recognition using the 3D visual sensor. Multiple cues, such as color, texture and 3D (depth) information, are integrated in order to recognize various types of objects under varying lighting conditions. We have implemented the system on our autonomous robot and made the robot do recognition tasks (object learning, detection, and recognition) in various environments. The results revealed that the proposed recognition system provides far better performance than the previous system that is based only on color and texture information.

In this paper, a novel object concept model, which encodes the relationship among appearance, functions and usage, is proposed. The essential attribute of an object (artifact) is its function that achieves a particular purpose. Therefore, the function model is constructed through observations from a camera at first. The function is defined as changes in the work object before and after tool use. At the same time, the usage model is constructed from observations of the hand shape, grasping parts, and contact points of the tool. And then, the proposed system learns the object concept that is based on the relationship among appearance, and learnt function and usage models. The object appearance is represented by SIFT (Scale Invariant Feature Transform). Since the proposed models are based on the graphical model, it is possible for the system to stochastically infer unobservable information from observed one. For example, the system can infer usage and/or functions of the tool visually through the proposed model. Some experimental results using the system, in which the proposed model is implemented, are shown to validate the proposed model.

Non-audible murmur (NAM) is an unvoiced speech signal that can be received through the body tissue with the use of special acoustic sensors (i.e., NAM microphones) attached behind the talker's ear. The authors had previously reported experimental results for NAM recognition using a stethoscopic and a silicon NAM microphone. Using a small amount of training data from a single speaker and adaptation approaches, 93.9% of word accuracy was achieved for a 20 k Japanese vocabulary dictation task. In this paper, further analysis of NAM speech is made using distance measures between hidden Markov models (HMMs). It has been shown that owing to the reduced spectral space of NAM speech, the HMM distances are also reduced when compared with those of normal speech. In the case of Japanese vowels and fricatives, the distance measures in NAM speech follow the same relative inter-phoneme relationship as that in normal speech without significant differences. However, significant differences have been found in the case of Japanese plosives. More specifically, in NAM speech, the distances between voiced/unvoiced consonant pairs articulated in the same place drastically decreased. As a result, the inter-phoneme relationship as compared to normal-speech changed significantly, causing a substantial decrease in the recognition accuracy. A speaker-dependent phoneme recognition experiment has been conducted, obtained 81.5% NAM phoneme correct, showing a relationship between HMM distance measures and phoneme accuracy. In a NAM microphone, body transmission and loss of lip radiation act as a low-pass filter. As a result, higher frequency components are attenuated in a NAM signal. Because of spectral reduction, NAM's unvoiced nature, and the type of articulation, NAM sounds become similar, causing a larger number of confusions when compared with normal speech. Yet many of those sounds are visually different on face/mouth/lips, and the integration of visual information increases their discrimination. As a result, recognition accuracy increases as well. In this article, the visual information extracted from the talkers' facial movements is fused with NAM speech. The experimental results reveal a relative improvement of 10.5% on average when fused NAM speech and facial information were used compared with using only NAM speech.

INTERSPEECH2009: 10th Annual Conference of the International Speech Communication Association, September 6-10, 2009, Brighton, UK.

In this paper we propose LDA-based framework for multimodal categorization and words grounding for robots. The robot uses its physical embodiment to grasp and observe an object from various view points as well as listen to the sound during the observing period. This multimodal information is used for categorizing and forming multimodal concepts. At the same time, the words acquired during the observing period are connected to the related concepts using multimodal LDA. We also provide a relevance measure that encodes the degree of connection between words and modalities. The proposed algorithm is implemented on a robot platform and some experiments are carried out to evaluate the algorithm. We also demonstrate a simple conversation between a user and the robot based on the learned model.

This paper discusses a computational model for object concept formation. We propose a model of object concept based on the relationship between shape and function. Implementation of the proposed framework using Bayesian network is presented. At this point we need an explicit definition of object function. In the proposed model each function is defined as certain changes in a target object caused by the object. Therefore each function is represented by a feature vector which quantifies the changes in the target. Then the function is abstracted from these feature vectors using the Bayesian learning approach. The system can form object concept by observing the human tool use based on the abstract function and shape information. Furthermore, it is demonstrated that the learned model (object concept) enables the system to infer the property of unseen object. The system is evaluated using 35 hand tools, which reveals the validity of the proposed framework.

This paper examines a simultaneous estimation of both high resolution (high accuracy) 3D information and high-resolution image from multiple low-resolution stereo-pair images. The proposed method is based on the MAP (maximum a posteriori) framework and these two relevant problems (estimation of both high-resolution 3 D and intensity images) are incorporated into a unified cost function. Then the solution is obtained iteratively by a fixed point algorithm, which yeilds 3 D information while the high-resolution image is fixed, and vice versa. Each solution can be obtained by solving a linear equation. It can be expected that the proposed method improves accuracy of both 3 D information and high-resolution image simultaneously. Some experimental results are shown to validate the proposed method.

In this paper unsupervised object categorization by robots is examined. We propose an unsupervised multimodal categorization based on audio-visual and haptic information. The robot uses its physical embodiment to grasp and observe an object from various view points as well as listen to the sound during the observation. The proposed categorization method is an extension of probabilistic Latent Semantic Analysis(pLSA), which is a statistical technique. At the same time the proposed method provides a probabilistic framework for inferring the object property from limited observations. Validity of the proposed method is shown through some experimental results.

In this paper, we describe the development of master-slave pneumatic robotic system and its ability to remotely control some functional tasks of backhoe machines during operation on construction sites. The functional tasks include (i) excavating (loading), (ii) lifting, (iii) transporting loads, and (iv) unloading. Pneumatic actuators and grippers, CCD cameras and a control box containing power source, PC and wireless LAN boards are integral part of our pneumatic robot system. During operation, images from the CCD cameras are transported to the master side with a wireless LAN, and then display visually on the laptop PC. Using joysticks, an operator can manipulate the synchronized activities of the system and backhoe machines with or without the CCD cameras. Tests were conducted on some local construction sites to evaluate the effectiveness of the system to control the functional tasks of the backhoe machines. It was found that the use of CCD cameras by the operator to monitor the synchronized activities was more effective than having a direct face to face operator contact with the activities. Disasters caused by environmental changes and man-made actions are increasing worldwide. The safety for rescued workers and investigative officials in disaster areas is a growing concern. Our system provides an opportunity to improve safety

In this paper, a novel approach to face-to-face transform is presented. The face-to-face transform is a technique, which transforms one person’s facial actions to the others. In gen-eral, the 3D models of faces are used for such transforma-tion. Therefore the facial action parameters should be es-timated from the 2D input images, which is not an easy task. On the contraly, our proposed approach is based on the 2D appearance model instead of the 3D model so that the model is acquired by learning directly from training im-ages. To achieve this, we investigate making use of the Hid-den Markov Model (HMM) framework, which models the correspondence between an input face and the other’s one as well as the appearances of both faces. The experimental results show the effectiveness of the proposed method. 1.

In this paper, a novel approach for wide band speech generation from narrow band is proposed. The proposed method is based on Subband Hidden Markov Model (Subband HMM). To train the HMM, a set of wide band speech is divided into a number of subbands and features are extracted independently. These extracted features are recombined and HMMs are trained by EM algorithm. The training process makes HMM to model the feature of signal in a single subband. In parallel, HMM learns the corresponding feature of all other subbands. The correspondence makes it possible to estimate the unobserved frequency component using the correspondence of narrowband signal to the HMM. We further investigate the application of the proposed method to denoising. Some experimental results are shown to confirm the validity of the proposed method.

The concept of 3-dimensional (3D) modeling by an autonomous mobile robot is presented. The mobile robot is equipped with a stereo camera and explores an unknown environment autonomously. The robot also has encoders and sonars. Information from these sensors are used for 3D modeling, ego-motion, and searching new and unknown objects. The 3D modeling of real environments by a mobile robot is applicable to optimal path planning, object recognition, virtual reality (VR), and so on. A preliminary result of the 3D modeling is also shown.

Previous work has shown that GenLOT coding is a very effective technique for compressing seismic data. The role of a transform in a coder is to concentrate information and reduce statistical redundancy. When used with embedded zerotree coding, GenLOTs often provide superior performance to traditional block oriented algorithms or to wavelets. In this work we investigate the use of generalized unequal length lapped orthogonal transforms (GULLOT). Their shorter bases for high-frequency components are suitable for reducing ringing artifacts in images. While GULLOTs yield comparable performance to GenLOTs on smooth seismic signals like stacked sections, they achieve improved performance on less smooth signals such as shot gathers.

A multi-channel speech separation method for real environments is proposed. The proposed method is based on frequency assignment, namely the magnitude of each channel at the same frequency bin is compared with each other and it is assigned to the channel, to which it originally belongs. This method is a direct consequence of frequency domain interpretation of the eigendecomposition method proposed by Cao et al., (1997). Furthermore, our proposed method does not require eigendecomposition, which consumes the costs of computation. We also present two example applications of the proposed method, that is, voice controlled computers in a multiuser environment and a noise removal in cellular telephony using two microphones.

This paper proposes a speech recognition and an enhancement system for noisy car environments based on a microphone array. In the system, multiple microphones axe arranged in 2-dimensional space, surrounding the interior of a car, and the speaker's location is first estimated by our proposed HE (harmonic enhanced) 2-D MUSIC (MUltiple SIgnal Classification). Then, 2-D delay and sum (DS) is applied to enhance the target speech. Such pre-processing makes robust speech recognition in noisy car environments possible. In the proposed system, not only a driver, but also a fellow passenger can control car electronics by their voices no matter where they are. This is an advantage of the system as well. The results of the simulation and the preliminary experiment in a real car environment are presented to confirm the validity of our proposed system.

In this paper, generalized linear phase lapped orthogonal transforms with unequal length basis functions (GULLOT) are considered. The length of each basis of the proposed GULLOT can be different from each other, while all the bases of the conventional GenLOT are of equal length. In order to apply the GULLOT to subband image coding, we also investigate the size-limited structure to process the finite length signal which is important in practice.

In this paper, we present optimal nonuniform lapped biorthogonal transforms for image compression. Subband coding using wavelet transforms have received a lot of attention because of multiresolution representation. The wavelet transform is regarded as a nonuniform filter bank and is constructed using the tree structure of a two-channel filter bank. However, it may not be optimal in the sense of energy compaction. If we can design nonuniform filter banks directly, it is possible to overcome the conventional wavelet transform. We then design a direct-structured nonuniform filter bank with a lapping window, called NULBT. Finally we show some examples and confirm the validity of our method.

Block orthogonal transforms (BOTs) are commonly used for lots of applications. Conventional BOTs are based on uniform filter banks, however, nonuniform BOTs are often superior to uniform ones. In this research, we investigate the factorization of nonuniform BOTs which does not involve the tree structure. Therefore, optimal nonuniform BOTs are available in the sense of transform coding gain. Some design examples are included to confirm our theory. We also apply the nonuniform BOT to the transform image coding.

In this research, we propose a direct design method of nonuniform filter banks (NUFBs). This method is based on frequency domain constraints to eliminate the amplitude and the aliasing distortions. Both NUFBs with integer and rational sampling factors can be designed with a common procedure. Here, we also consider the design method which requires only to solve linear equations iteratively. In our proposed method, least square errors of the perfect reconstruction (PR) constraints are minimized without using the nonlinear programming technique.

In this paper, we propose a design method of cosine-modulated 2-dimensional (2-D) perfect reconstruction (PR) FIR filter banks, in which each analysis and synthesis filter has linear phase (LP). As in previous works, it is impossible to design the 2-D cosine modulated system which satisfies both the PR condition and LP property. However, in 2-D signal processing, it is very important for each filter to have the LP property. Our main goals are to show such a system can be realized by considering a new decomposition scheme and derive the conditions for PR and LP. A design example is included to show the effectiveness of this paper.