Many people, especially Japanese, are embarrassed to converse with agents such as virtual assistants, probably due to a low social presence, which refers to the degree to which one perceives the human-like properties of an agent. We assumed that poor emotional expressions of agents may impair their human-likeness. In this study, we proposed melodic emotional expression (MEE), which is a new auditory emotional expression for spoken dialog agents. We added background music (BGM) and sound effects as MEE to synthetic voices and conducted experiments to investigate the effects. First, we found that adding MEE to the flat synthetic voice could convey emotions as intended. We also found that when positive emotions were expressed by MEE, it made the agent more human-like and easier to talk to. Furthermore, we achieved these effects when MEE was added to an emotional synthetic voice. These effects were particularly noticeable with the BGM. We further attempted automatic BGM generation, which is necessary for the practical application of MEE. Listeners accurately categorized the BGM generated by the prototype system into four types of emotions: joy, angry, sad, and relaxed.

Combining a device that mediates social touch between people in remote locations with a video call can be expected to enhance the feeling of being in the same space (sense of space sharing) and social bonding with the remote partner. However, when the device is combined with a mobile phone video call, size inconsistency could occur between the device that acts as a substitute for the body part and the reduced video of the partner. In addition, when the device operates separately from a mobile phone in wireless communication, positional inconsistency also happens. This study developed a robotic hand-type social touch device and investigated whether it enhances the sense of space sharing and social bonding even though these inconsistencies occur. Experimental results revealed that the user's belief that the finger movements of the device were the partner's hand movement contributed to the sense of space sharing. The movements of the device according to the flow of conversation facilitated this belief; hence, the sense of space sharing increased in spite of the inconsistencies. Positional inconsistency was also found to have an adverse effect on social bonding. Positional consistency contributes to enhance social bonding because it visually facilitates the user's belief that the device is a substitute for the partner's body. Basically, the functions of the device, such as skin softness, body temperature, finger movements, and grip force, which enhance the reality of the touch sensation, could be effective in facilitating the user's beliefs. However, it was suggested that awareness of temperature does not always make positive contribution to the reality and could lead to eeriness.

There are many researches that devise the appearance of agents that recommend products, but there are not many researches that focus on the environment in which a robot and a person interact. In this research, we examined whether there are differences in the impression on the robot and the change in the degree of interest in the product under the face-to-face condition and the 90° condition when the robot explains the product. As a result, there was no significant difference in the degree of interest in the products between the two conditions, but in the evaluation of the impression on the robot, it was found that the 90° condition was more likely to be felt as 'obedient" and "humorous" significantly. Therefore, it was found that although the 90° condition gives a better impression on the robot than the face-to-face condition, it does not affect the degree of interest in the product explained by the robot.

evel (Austin 1962). DAs constitute the most fundamental part of communication, and the comprehension of DAs is essential to human-agent interaction. The purpose of this study is to enable an agent to behave properly in response to DAs without their explicit representation on one hand and to estimate the DAs explicitly on the other hand. The former is realized by reinforcement learning and the latter by Bayesian inference. The simulation results demonstrated that the agent not only responded to DAs successfully but also inferred the DAs correctly.

Authors have conducted an observation experiment mainly on conversational play for creating the basic findings of childcare support by smart speakers. Our previous study analyzed children's attitudes and behaviors toward an utterance-output device during conversational play. In this experiment, the mother was beside her child, and the influence of the mother on the conversational play seemed to be great, but we have not been analyzed it yet. In this study, we analyzed the relationships between traits and behaviors of mothers, and children's conversational play. The results suggest that the more epistemic curiosity mothers are, the more children follow the instruction of an utterance-output device as third person, and that the more mothers prompt the conversational play toward their children, the less they react to phrases of the utterance-output device.

Dialogue act (DA) represents the meaning of an utterance at the illocutionary force level (Austin 1962) such as questions, requests, and greetings. Since DAs take charge of the most fundamental part of communication, we believe that the elucidation of DA learning mechanism is important for cognitive science and artificial intelligence, especially for cognitive robotics. The purpose of this study is to verify that scaffolding takes place when a human teaches a robot, and to let a robot learn to estimate DAs and to make a response based on them step by step utilizing scaffolding provided by a human. To realize that, it is necessary for the robot to detect changes in utterance and rewards given by the partner and continue learning accordingly. Experimental results demonstrated that participants who continued interaction for a sufficiently long time often gave scaffolding for the robot. Although the number of experiments is still insufficient to obtain a definite conclusion, we observed that 1) the robot quickly learned to respond to DAs in most cases if the participants only spoke utterances that match the situation, 2) in the case of participants who built scaffolding differently from what we assumed, learning did not proceed quickly, and 3) the robot could learn to estimate DAs almost exactly if the participants kept interaction for a sufficiently long time even if the scaffolding was unexpected.

A dialogue act (DA) represents the meaning of an utterance at the illocutionary force level (Austin 1962) such as questions, requests, and greetings. Since DAs take charge of the most fundamental part of communication, we believe that the elucidation of DA learning mechanism is important for cognitive science and artificial intelligence. The purpose of this study is to let a robot learn to estimate DAs and to make a response based on them and to verify that scaffolding takes place when people teach the robot. The experimental results demonstrated that participants who continued interaction for a sufficiently long time gave scaffolding for the robot.

The purpose of this paper is to examine the replicability, generalizability, and efficacy of the optimal rate of mimicry in human-robot interactions reported by Y. Shinohara et al. (HAI'16, 2016). That study showed that a mimicry rate of about 83% was optimal to increase the likability of an upper-body humanoid Nao robot. To replicate the findings regarding the optimal rate of mimicry, and to examine these generalizability of the results, in study 1 we conducted experiments using the same method with two kinds of humanoid robots: a whole-body humanoid Nao robot and a whole-body humanoid Nao robot with randomly changing eye color between white to red. In study 2, we investigated whether increased likability triggered by mimicry also enhances helping behaviors from a human participant. Our study results confirmed the earlier findings from Shinohara et al. reinforcing the finding about the optimal rate of mimicry in human-robot interaction. Furthermore, we could demonstrated that mimicry can lead to helping behavior from the human participant which could have important implications for establishing cooperative interactions between humans and robots.

In most role-playing games, a non-player character (NPC) tends to have the most initiative, i.e., developing conversation topics, on the dialog. Thus, the player tends to experience a unilateral impression, i.e., the feeling of that an NPC forces conversation topics on the player in spite of the player's willingness, and gradually loses his/her motivation to communicate with an NPC. In order to address this problem, this study proposes a dialog method whereby an NPC can adaptively switch the initiative by determining the player's willingness to talk/listen to the NPC. Based on the results of an evaluation experiment in which the participants talked with the NPC in the game scenario, it is suggested that the aspect of mixed-initiative in our proposed method could decrease the unilateral impression associated with the existing method that an NPC has the most initiative. Furthermore, the determination of the player's desire in our proposed method could further motivate players to talk to the NPC.

The importance of building rapport between a human and an agent is increasing with the burgeoning development of robot technology. Several recent studies have focused on the chameleon effect, using psychological concepts to investigate human-agent interaction. However, the validity of the chameleon effect in human-agent interaction is controversial. Few studies have explored the influence of individual cognitive ability and the rate of mimicry on the human-agent interaction. We explored the optimal rate of mimicry and the relationship between mimicry rate and individual empathic ability. We controlled the amount of agent mimicry and examined the effect on participants classified as high- and low-perspective takers. We found that, overall, participants preferred agents that mimicked their behavior 83% of the time. Moreover, high-, but not low-, perspective takers tended to be influenced by the mimicry rate.

Studies have shown that a person's current body state can affect his thinking (embodied cognition). We tested how value judgments can be changed by this. The results of our experiment demonstrated that participants tended to discount a future reward less when looking up than when looking down. Moreover, we found that the β parameter, which represents the value of immediate rewards relative to delayed rewards received at another point in time, significantly differed between the two conditions; whereas the δ parameter, the discount rate in the standard exponential formula, did not show a significant difference. Using functional magnetic resonance imaging, McClure et al. (2004) showed that β is mediated by the lower level, automatic processes of the limbic structures; δ is mediated by the lateral prefrontal cortex, supporting higher cognitive functions. Combining the above two results, we can conclude that the embodied cognition in our experiment was mainly produced by the lower level brain processes. We believe that the knowing that discount factor β can be controlled by posture can be applied when designing robot behavior, such as encouraging a diet, trying to sell insurance, or offering customers a card loan.

It is important that participants form a positive impression when they meet with a new robot for the first time. Therefore, we tried to change how participants evaluate a new robot by changing the surrounding environment in the experiment. Embodied Cognition is a theory that our cognition for a target is strongly influenced by physical stimulation of our own bodies. Our research investigated a hypothesis that a soft chair can make favorable impressions of a robot. We performed an experiment using a hard plastic chair and a soft cushioned chair. Although the results did not support the hypothesis, they suggested that the effect of embodied cognition can be different between males and females.

Intimacy is a very important factor not only for the communication between humans but also for the communication between humans and robots. In this research we propose an action decision model based on others' friendliness value, which can be estimated using the mental models of others. We examine the mutual adaptation process of two agents, each of which has its own model of others, through the interaction between them.

It is difficult to design robotic playmates for introverted children. Therefore, we examined how a robot should play with such shy children. In this study, we hypothesized and tested an effective play strategy for building a good relationship with shy children. We conducted an experiment with 5- to 6-year-old children and a humanoid robot teleoperated by a preschool teacher. We developed a valid play strategy for shy children.

Similar to relationships between humans, a person desiring to form a good relationship with a robot needs to be able to empathize with it. However, the specific kinds of human-robot interactions that would arouse and enhance empathy for the robot in the user's mind have not yet been clarified. In addition, the human behavioral traits that may be regarded as indices of empathy have not been investigated extensively. In an attempt to address these two issues, a preliminary experiment on empathy in human-robot interaction is conducted. The results suggest that the actions of naming or comforting a robot could contribute to enhancing its user's empathy and that eye fixation could be used as an index of empathy even when the use of a subjective index is inconclusive.

The task of automated risk assessment is attracting significant attention in the light of the recent microloan popularity growth. The industry requires a real time method for the timely processing of the extensive number of applicants for short-term small loans. Owing to the vast number of applications, manual verification is not a viable option. In cooperation with a microloan company in Azerbaijan, we have researched automated risk assessment using crowdsourcing. The principal concept behind this approach is the fact that a significant amount of information relating to a particular applicant can be retrieved from the social networks. The suggested approach can be divided into three parts: First, applicant information is collected on social networks such as LinkedIn and Facebook. This can only occur with the applicant's permission. Then, this data is processed using a program that extracts the relevant information segments. Finally, these information segments are evaluated using crowdsourcing. We attempted to evaluate the information segments using social networks. To that end, we automatically posted requests on the social networks regarding certain information segments and evaluated the community response by counting “likes” and “shares”. For example, we posted the status, “Do you think that a person who has worked at ABC Company is more likely to repay a loan? Please “like” this post if you agree.” From the results, we were able to estimate public opinion. Once evaluated, each information segment was then given a weight factor that was optimized using available loan-repay test data provided to us by a company. We then tested the proposed system on a set of 400 applicants. Using a second crowdsourcing approach, we were able to confirm that the resulting solution provided a 92.5% correct assessment, with 6.45% false positives and 11.11% false negatives, with an assessment duration of 24 hours.

Learning is essential for an autonomous agent to adapt to an environment. One method that can be used is learning through trial and error. However, it is impractical because of the long learning time required when the agent learns in a complex environment. Therefore, some guidelines are necessary to expedite the learning process in a complex environment. Imitation of the behavior of other agents who have already adapted to the environment would shorten an agent's learning time. Thus, imitation can be used by agents as a guideline for learning. In this study, we propose a computational model of imitation and autonomous behavior. We expect that an agent can reduce its learning time through imitation. The actions that an agent performs are represented by a set of features such as the type, location, and object of an action. The agent tends to imitate the similar actions of other agents, and the similarity between actions is calculated, which is indicative of the importance of each feature. The proposed model is evaluated using a dining table simulator. The experimental results indicate that the proposed model can adapt to the environment faster than a baseline model that learns only through trial and error, and that the proposed model can shorten the learning time further if the importance of each feature can be adjusted by learning.

When talking to infants, we tend to use infant-directed speech (IDS) rather than adult-directed speech (ADS). IDS attracts more attention from infants, conveys the emotions of adults more easily, and makes language acquisition easier. It is not clear, however, whether IDS has a cognitive effect on adults as well. To address this issue, we focus on one of the most distinctive features of IDS, a high-pitched voice. In addition, we conduct two human-robot interaction experiments to examine the following two hypotheses: (i) a robot reacting selectively to a high-pitched voice triggers a high-pitched voice of the user (H1), and (ii) talking to a robot in a high-pitched voice improves the user's impression of the robot (H2). The results did not support H1, but marginally supported H2.

Multi-Agent Systems have been in the focus of research interest for a long time. These systems are used widely in such areas as logistics, management, various simulations, game development and many others. In this paper, we search for a method to increase flexibility and efficiency of Multi-Agent Systems by investigating the usefulness of Decision Support Systems for intelligent agents in multi-agent environment. We also research advantages, such as the ability to add new agents at run-time, and disadvantages, such as increased complexity, of creating and modifying Decision Support Systems dynamically in Multi-Agent System.

If robots learn new actions through human-robot interaction, it is important that the robots can utilize rewards as well as instructions to reduce humans' efforts. Additioanlly, "interval" which allows humans to give instructions and evaluations is also important. We hence focused on "delays in initiating actions" and changed them according to the progress of learning: long delays at early stages, and short at later stages. We compared the proposed varying delay with a constant delay by an experiment. The result demonstrated that the varying delay improves learning efficiency significantly and impresses humans as teachable.

In this paper, we propose a novel method to detect robot-directed (RD) speech that adopts the Multimodal Semantic Confidence (MSC) measure. The MSC measure is used to decide whether the speech can be interpreted as a feasible action under the current physical situation in an object manipulation task. This measure is calculated by integrating speech, image, and motion confidence measures with weightings that are optimized by logistic regression. Experimental results show that, compared with a baseline method that uses speech confidence only, MSC achieved an absolute increase of 5% for clean speech and 12% for noisy speech in terms of average maximum F-measure.

In this paper, we propose a novel method for a robot to detect robot-directed speech, that is, to distinguish speech that users speak to a robot from speech that users speak to other people or to themselves. The originality of this work is the introduction of a multimodal semantic confidence (MSC) measure, which is used for domain classification of input speech based on the decision on whether the speech can be interpreted as a feasible action under the current physical situation in an object manipulation task. This measure is calculated by integrating speech, object, and motion confidence with weightings that are optimized by logistic regression. Then we integrate this measure with gaze tracking and conduct experiments under conditions of natural human-robot interaction. Experimental results show that the proposed method achieves a high performance of 94% and 96% in average recall and precision rates, respectively, for robot-directed speech detection.

This paper describes a novel method that enables users to teach systems the phoneme sequences of new words through speech interaction. Using the method, users can correct mis-recognized phoneme sequences incrementally by making corrective utterances. Each corrective utterance may include the whole or a segment of the word. During the interaction, if the correction using the utterance results in a better phoneme sequence than the previous one, a user can stop the interaction or make a corrective utterance again. Otherwise the user can reject the utterance. The originalities of this method are 1) interactive correction by speech, 2) the use of spoken word segments for locating mis-recognized phonemes and, 3) the use of generalized posterior probability (GPP) as a measure of correcting mis-recognized phonemes. The experimental results show that the proposed method achieved 96.8% in phoneme accuracy and 79.1% in word accuracy, with less than seven corrective utterances.

Interactive agents such as pet robots or adaptive speech interface systems that require forming a mutual adaptation process with users should have two competences. One of these is recognizing reward information from users' expressed paralanguage information, and the other is informing the learning system about the users by means of that reward information. The purpose of this study was to clarify the specific contents of reward information and the actual mechanism of a learning system by observing how 2 persons could create a smooth speech communication, such as that between owners and their pets. A communication experiment was conducted to observe how human participants create smooth communication through acquiring meaning from utterances in languages they did not understand. Then, based on experimental results, a meaning-acquisition model that considers the following 2 assumptions was constructed: (a) To achieve a mutual adaptive relationship with users, the model needs to induce users' adaptation and to exploit this induced adaptation to recognize the meanings of a user's speech sounds; and (b) to recognize users' utterances through trial-and-error interaction regardless of the language used, the model should focus on prosodic information in speech sounds, rather than on the phoneme information on which most past interface studies have focused. The results confirmed that the proposed model could recognize the meanings of users' verbal commands by using participants' adaptations to the model for its meaning-acquisition process. However, this phenomenon was observed only when an experimenter gave the participants appropriate instructions equivalent to catchphrases that helped users learn how to use and interact intuitively with the model. Thus, this suggested the need for a subsequent study to discover how to induce the participants' adaptations or natural behaviors without giving these kinds of instructions.