Leveraging electrocardiogram (ECG) signals for emotion recognition represents a core challenge in affective computing, particularly in achieving robustness across diverse demographic groups (such as older adults with mild cognitive impairment). This challenge is rooted in three key issues: the complex multi-scale nature of ECG signals, high inter-individual physiological variability, and the need for computationally efficient temporal modeling for IoT applications. To address these issues systematically, this study proposes HCMNet, a novel, physiologically-inspired hybrid Convolutional-Mamba network. HCMNet’s architecture is problem-driven: a hierarchical scale-aware convolutional module captures multi-scale features analogous to HRV analysis; an innovative Non-Local Channel Convolutional Attention (NLCCA) mechanism mitigates inter-individual variability by learning to reshape the feature space; and a Mamba2-based Bidirectional State-Space Model (BiSSM) efficiently models temporal dynamics with linear complexity. Additionally, we validated the model on a self-built Wearable ECG emotion dataset comprising healthy elderly individuals and patients with mild cognitive impairment (MCI), as well as on public datasets WESAD and DREAMER. Experimental results demonstrate that our proposed HCMNet, through its synergistic hybrid architecture, effectively extracts robust emotional features. It not only achieves state-of-the-art performance on public benchmarks but also exhibits strong robustness for special populations. Furthermore, our in-depth adaptation analysis reveals that while a “one-model-fits-all” approach is infeasible for unseen subjects, HCMNet excels as a robust transferable base model that can be rapidly personalized, offering a practical paradigm for accurate and adaptable emotion recognition in real-world IoT settings. The source code is available at https://github.com/INSOCE/HCMNet.

Exploring the interaction dynamics between humans and robots also entails examining the exchange of multisensory signals. In this light, this study explores the role of social odors (putative pheromones) in shaping human-robot interactions (HRI) beyond the conventional focus on visual and auditory cues. Our research examines how these odors influence relational engagement (RE; assessed as co-presence and social presence) and trust (dispositional and situational) during interactions with the humanoid social robot (HSR) NAO. The robot was also equipped with a male or female voice, to better counteract odor-related gender differences. Thirty participants engaged in storytelling sessions with NAO under different sensory conditions. Our findings indicate odors significantly affecting trust, especially when resembling female pheromones. Conversely, odors had a nuanced influence on RE, despite participants reporting greater CoPre than SoPre. Additionally, a bidirectional relationship between SoPre and trust emerged, reinforced by sensory congruence. These results highlight the significance of sensory alignment in shaping socio-relational and trust dynamics and emphasize the potential of olfactory stimuli in enhancing human-robot interactions. By deepening our understanding of the complex interplay among olfactory cues, voice gender, RE, and trust within HRIs, we provide valuable insights for crafting and deploying HSRs that are more relatable and trustworthy.

Providing a robot with the capabilities of understanding and effectively adapting its behaviour based on human mental states is a critical challenge in Human–Robot Interaction, since it can significantly improve the quality of interaction between humans and robots. In this work, we investigate whether considering human mental states in the decision-making process of a robot improves the transparency of its behaviours and mitigates potential human’s false beliefs about the environment during collaborative scenarios. We used Bayesian inference within a Hierarchical Reinforcement Learning algorithm to include human desires and beliefs into the decision-making processes of the robot, and to monitor the robot’s decisions. This approach, which we refer to as Hierarchical Bayesian Theory of Mind, represents an upgraded version of the initial Bayesian Theory of Mind, a probabilistic model capable of reasoning about a rational agent’s actions. The model enabled us to track the mental states of a human observer, even when the observer held false beliefs, thereby benefiting the collaboration in a multi-goal task and the interaction with the robot. In addition to a qualitative evaluation, we conducted a between-subjects study (110 participants) to evaluate the robot’s perceived Theory of Mind and its effects on transparency and false beliefs in different settings. Results indicate that a robot which considers human desires and beliefs increases its transparency and reduces misunderstandings. These findings show the importance of endowing Theory of Mind capabilities in robots and demonstrate how these skills can enhance their behaviours, particularly in human–robot collaboration, paving the way for more effective robotic applications.

Multimodal Emotion Recognition in Conversation (MERC) is an important element in human-machine interaction. It allows machines to automatically identify and track the emotional status of speakers during a conversation in a multimodal setting. However, the conversations involving various audio and visual cues aligned with textual cues are very complex. Recent works have tried integrating the audio and visual modalities with textual to improve the performance of emotion recognition in conversation. Although many MERC models leverage textual, audio, and visual modalities, those models assume that the speaker’s textual utterance, audio speech, and facial sequences are present. However, a conversation may contain multiple parties, among which only one is the speaker. Previous MERC assumed the availability of all modalities, but in many instances, one or more modalities may be unavailable during multiparty conversations. To tackle these issues, we propose the Possible Speaker Informed Multimodal Emotion Recognition in Conversation framework (PSI). PSI is specifically tasked to extract audio (speech) and visual (face) sequences of a possible speaker in the presence of multiple parties. Further, PSI seamlessly extracts the rich unimodal features and fuses them while addressing the unavailability of specific modalities. PSI demonstrates competitive performance with existing state-of-the-art models through experiments with a benchmark dataset.

Human-robot collaboration has great potential in enhancing robot deployment at close proximity with people, especially in non-dyadic collaborations with multiple users. However, autonomous systems that are capable of handling such interactions in a physical domain are rare. This work proposes TriHRCBot, a robotic architecture designed to handle a collaborative task that involves two concurrent users. The architecture is sensitive to position, orientation, body lengths and state of the users in the interaction, and uses this information to adjust the pose of a target object to enable both users to act on it at the same time. A robotic system equipped with the TriHRCBot architecture was deployed in a user study in which 30 participants from the BAE Systems Academy for Skills and Knowledge Centre interacted with it during such multi-user collaborative task. The study shows that the participants considered TriHRCBot acceptable for the task at hand. The code repository of the architecture is publicly available11Code repository: https://github.com/francescosemeraro/TriHRCBot,

Current approaches in Explainable Deep Reinforcement Learning have limitations in which the attention mask has a displacement with the objects in visual input. This work addresses a spatial problem within traditional Convolutional Neural Networks (CNNs). We propose the Interpretable Feature Extractor (IFE) architecture, aimed at generating an accurate attention mask to illustrate both "what" and "where" the agent concentrates on in the spatial domain. Our design incorporates a Human-Understandable Encoding module to generate a fully interpretable attention mask, followed by an Agent-Friendly Encoding module to enhance the agent’s learning efficiency. These two components together form the Interpretable Feature Extractor for vision-based deep reinforcement learning to enable the model’s interpretability. The resulting attention mask is consistent, highly understandable by humans, accurate in spatial dimension, and effectively highlights important objects or locations in visual input. The Interpretable Feature Extractor is integrated into the Fast and Data-efficient Rainbow framework, and evaluated on 57 ATARI games to show the effectiveness of the proposed approach on Spatial Preservation, Interpretability, and Data-efficiency. Finally, we showcase the versatility of our approach by incorporating the IFE into the Asynchronous Advantage Actor-Critic Model.

An agent’s intention often remains hidden behind the black-box nature of embodied policies. Communication using natural language statements that describe the next action can provide transparency towards the agent’s behavior. We aim to insert transparent behavior directly into the learning process, by transforming the problem of policy learning into a language generation problem and combining it with traditional autoregressive modeling. The resulting model produces transparent natural language statements followed by tokens representing the specific actions to solve long-horizon tasks in the Language-Table environment. Following previous work, the model is able to learn to produce a policy represented by special discretized tokens in an autoregressive manner. We place special emphasis on investigating the relationship between predicting actions and producing high-quality language for a transparent agent. We find that in many cases both the quality of the action trajectory and the transparent statement increase when they are generated simultaneously.

Nonverbal communication plays a crucial role in both human-human and human-robot interactions (HRIs), where facial expressions convey emotions, intentions and trust. Enabling humanoid robots to generate human-like facial reactions in response to human speech and facial behaviours remains significant challenges. In this work, we leverage human-human interaction (HHI) datasets to train a humanoid robot, allowing it to learn and imitate facial reactions to both speech and facial expression inputs. Specifically, we extend a sequence-to-sequence (Seq2Seq)-based framework that enables robots to simulate human-like virtual facial expressions that are appropriate for responding to the perceived human user behaviours. Then, we propose a deep neural network-based motor mapping model to translate these expressions into physical robot movements. Experiments demonstrate that our facial reaction–motor mapping framework successfully enables robotic self-reactions to various human behaviours, where our model can best predict 50 frames (two seconds) of facial reactions in response to the input user behaviour of the same duration, aligning with human cognitive and neuromuscular processes. Our code is provided at https://github.com/mrsgzg/Robot_Face_Reaction.

As hybrid interactions between humans and artificial agents become more prevalent, social skills are increasingly essential for autonomous systems. Beyond assisting in various tasks, robots are expected to understand human states and recognize that knowledge and perceptions of the world can differ, influencing overall behavior. This ability is closely tied to motivation, which plays a crucial role in driving autonomous agents’ actions. In this work, we explore the interaction between two intrinsically motivated cognitive autonomous robots with distinct profiles and preferences, utilizing Theory of Mind to infer each other’s motivations. We investigate the conditions under which they successfully collaborate to achieve mutual well-being and the circumstances that hinder cooperation. Our findings indicate that successful interactions emerge when at least one agent prioritizes helping others and when their profiles are aligned, leading to positive outcomes for both.

For social robots to collaborate effectively, they must infer and correct false human beliefs, especially when misconceptions directly impact task outcomes or pose safety risks to humans. In this work, we investigated whether a robot’s ability to detect and rectify users’ false beliefs improves trust and perceived social competence. In an in-person between-subject study, 98 participants collaborated with a robot to solve a task. Participants interacted with either a robot that actively corrected their false beliefs by using Theory of Mind or one that complied with their incorrect instructions. Contrary to expectations, trust, mental state attribution, and perceived warmth or competence did not differ between groups. The results also showed that human reluctance to trust the robot’s input persisted, suggesting that belief correction alone cannot overcome relational barriers. In addition, the study showed that participants who trusted the robot’s corrections perceived it as more socially attuned.

The advancement of collaborative robots emphasizes fast, efficient, and adaptive learning, particularly in environments with visual occlusions. While deep learning has proven effective in robotics, its reliance on large datasets, high computational resources, and limited interpretability poses significant challenges. To address these issues, we explored the use of the Compact Prediction Tree as an efficient and explainable machine learning approach for sequential pattern recognition. We used this algorithm in a human-robot collaboration scenario, where a robot assisted a user to assemble a cubic scaffold by handing over the correct assembly pieces, even under partial visibility of the workspace. Experimental validations showed that the system completed the task perfectly when a single component of the assembly was occluded, and maintained an average task completion rate of at least 0.8 when two components were occluded, even when trained on only 20% of the dataset.The code repository of the robotic system is publicly available1.

With the rapid proliferation of the Internet of Things (IoT), autonomous vehicles (AVs), or self-driving cars, rely heavily on real-time data sharing and message exchanges over wireless networks. AVs use sensors, artificial intelligence, machine learning, and advanced algorithms to perform various functions, enabling users to operate without human intervention. Owing to the flexibility and high mobility of drones, they could aid in the operations of AVs. However, the security and privacy are the main concerns; specifically, the threat of physical capture and violation of anonymity are the main hurdles for realization of secure communication among the AVs and drones. To address these challenges, we propose an anonymous and provably secure lightweight authentication protocol for unmanned-aerial-vehicle-assisted Internet of Autonomous Vehicles (SLAP-IoAV). The proposed protocol uses cryptographic primitives such as exclusive-OR operations, elliptic-curve cryptography, collision-resistant one-way hashing, and concatenation to ensure robust security. An informal security analysis found that SLAP-IoAV is secure against several known attacks, and a performance analysis established that the protocol has less computational and communication overhead than existing competitive protocols. Additionally, Scyther simulation results confirm that no security vulnerabilities are present. Overall, our protocol delivers superior security and performance, making it well-suited to real-world applications in the AV industry.

The combination of wearable sensors and competitive sports provides quantitative information for scientific training, effectively assisting athletes in improving their athletic performance. This study presents a technical framework for athletic sports assessment in competitive swimming based on body-area sensor networks. In our approach, wearable inertial sensor nodes are placed on specific body parts of the athletes to capture motion data during different competitive swimming strokes. Multiwearable inertial sensor nodes are worn on specific body parts of athletes for real-time monitoring motion data during training sessions. A motion intensity detection-based error-state-Kalman-filter algorithm is proposed for multisensor data fusion. Additionally, through kinematic statistical analysis, the characteristics of joint motion during training are clearly explained. Furthermore, a deep learning network that fuses sensor time series and human skeleton graphs is proposed for different stroke phase segmentation, enabling quantitative measurement of motion phases, and several baseline classifiers are chosen for comparison to validate the robustness of our phase segmentation method. We also investigate the sensor combination selection issue during the phase segmentation process to determine the optimal sensor configuration. Our approach provides a scientific solution for the integration of wearable sensors and competitive sports, contributing to the high-quality development of the next generation of smart sports.

Gait phase segmentation, pivotal for understanding lower limb motion, finds applications in diverse fields like medicine and sports. While existing method often struggle with accuracy and adaptability in real-world settings, this study presents a novel methodology employing particle filters for precise lower limb motion capture (MoCap) utilizing inertial sensors, which can be used in more everyday environments and in a wider range of applications over a longer period of time. The innovative approach adeptly tracks walking movements, labeling six gait phases via skeleton reconstruction facilitated by the MoCap algorithm. Subsequently, we propose a neural network architecture amalgamating temporal convolutional network (TCN), graph convolutional network (GCN), and long short-term memory (LSTM). This architecture integrates raw data from inertial sensors with joint angles derived from reconstructed motion, achieving accurate segmentation of the six gait phases. Experimental validation compares the MoCap algorithm against an optical motion capture system, and the neural network’s performance against state-of-the-art methods. Results demonstrate our method’s superior accuracy of 96.94%, highlighting its efficacy in addressing gait phase segmentation challenges and propelling advancements in gait analysis.

Facial Expression Recognition (FER) models based on deep learning mostly rely on a supervised train-once-test-all approach. These approaches assume that a model trained on an in-the-wild facial expression dataset with one type of domain distribution will perform well on a test dataset with a domain distribution shift. However, facial images in real-world can be from different domain distributions from which the model has been trained. However, re-training models on only new domain distributions will severely affect the performance of the previous domain. Re-training on all previous and new data can improve overall performance but is computationally expansive. In this study, we oppose the train-once-test-all approach and propose a buffer-based continual learning approach to enhance the performance of multiple in-the-wild datasets. We propose a model that continually leverages attention to important facial features from the pre-trained model to improve performance in multiple datasets. We validated our model using split-in-the-wild datasets where the dataset is provided to the model in an incremental setting instead of all at once. Furthermore, to evaluate the model performance, we continually used three in-the-wild datasets representing different domains (Domain-FER). Extensive experiments on these datasets reveal that the proposed model achieves better results than other Continual FER models.

Trust is essential for social interactions, including those between humans and social artificial agents, such as robots. Several factors and combinations thereof can contribute to the formation of trust and, importantly in the case of machines that work with a certain margin of error, to its maintenance and repair after it has been breached. In this article, we present the results of a study aimed at investigating the role of robot voice and chosen repair strategy on trust formation and repair in a collaborative task. People helped a robot navigate through a maze, and the robot made mistakes at pre-defined points during the navigation. Via in-game behaviour and follow-up questionnaires, we could measure people’s trust towards the robot. We found that people trusted the robot speaking with a state-of-the-art synthetic voice more than with the default robot voice in the game, even though they indicated the opposite in the questionnaires. Additionally, we found that three repair strategies that people use in human-human interaction (justification of the mistake, promise to be better and denial of the mistake) work also in human-robot interaction.

Although attention mechanisms have achieved considerable progress in Transformer-based architectures across various Artificial Intelligence (AI) domains, their inner workings remain to be explored. Existing explainable methods have different emphases but are rather one-sided. They primarily analyse the attention mechanisms or gradient-based attribution while neglecting the magnitudes of input feature values or the skip-connection module. Moreover, they inevitably bring spurious noisy pixel attributions unrelated to the model’s decision, hindering humans’ trust in the spotted visualization result. Hence, we propose an easy-to-implement but effective way to remedy this flaw: Smooth Noise Norm Attention (SNNA). We weigh the attention by the norm of the transformed value vector and guide the label-specific signal with the attention gradient, then randomly sample the input perturbations and average the corresponding gradients to produce noise-free attribution. Instead of evaluating the explanation method on the binary or multi-class classification tasks like in previous works, we explore the more complex multi-label classification scenario in this work, i.e., the driving action prediction task, and trained a model for it specifically. Both qualitative and quantitative evaluation results show the superiority of SNNA compared to other SOTA attention-based explainable methods in generating a clearer visual explanation map and ranking the input pixel importance.

Theory of Mind (ToM) is a fundamental cognitive architecture that endows humans with the ability to attribute mental states to others. Humans infer the desires, beliefs, and intentions of others by observing their behavior and, in turn, adjust their actions to facilitate better interpersonal communication and team collaboration. In this paper, we investigated trust-aware robot policy with the theory of mind in a multi-agent setting where a human collaborates with a robot against another human opponent. We show that by only focusing on team performance, the robot may resort to the reverse psychology trick, which poses a significant threat to trust maintenance. The human’s trust in the robot will collapse when they discover deceptive behavior by the robot. To mitigate this problem, we adopt the robot theory of mind model to infer the human’s trust beliefs, including true belief and false belief (an essential element of ToM). We designed a dynamic trust-aware reward function based on different trust beliefs to guide the robot policy learning, which aims to balance between avoiding human trust collapse due to robot reverse psychology and leveraging its potential to boost team performance. The experimental results demonstrate the importance of the ToM-based robot policy for human-robot trust and the effectiveness of our robot ToM-based robot policy in multiagent interaction settings.

Human-robot collaboration has the potential of unlocking new manufacturing paradigms thanks to the introduction of a robotic architecture in a production chain that involves human workers. A possible innovative declination of this is the use of collaborative robots to enable two workers to concurrently act on the same manufacturing target without causing mutual disturbances. By doing so, the efficiency of the process would be preserved while reducing the production times. This work designs a physical collaborative task that involves two users and one collaborative robot. The users act in the scenario in a concurrent way on the same target object, while the robot physically intervenes in the scene as a mediator by adjusting the position and orientation of the object to accommodate both users at the same time. Through this experimental setup, 78 apprentices and teachers of the BAE Systems Academy for Skills and Knowledge Centre were recruited to investigate the users’ perception of the task workload and trust towards the robotic system. Specifically, they performed the same task under two experimental conditions, in which the robot responded to changes in the interaction in a reactive or timed way, respectively. The statistical analysis showed that a timed response of the robot was associated with lower perceived workload and higher predictability of the system.

Understanding human emotions is paramount for effective human-human interactions. As technology advances, social robots are increasingly being developed with the capability to discern and respond to human emotions, with the ultimate aim of providing assistance and companionship. However, existing research on emotion recognition for human-robot interaction predominantly focuses on facial expressions or verbal speech, neglecting other potential mediums of emotional expression. In this study, we shed light on the significance of considering various forms of emotional expression, mainly nonverbal vocalization known as vocal bursts, which have been overlooked in emotion modeling for human-robot interaction. Vocal bursts, characterized by brief and intense vocal utterances, represent a rich source of emotional cues that can significantly enhance the capabilities of social robots in understanding and responding to human emotions. Driven by the increasing interest in vocal bursts within speech and affective computing research, we propose a baseline model for affective vocal burst recognition that can outperform large audio models. The proposed baseline model achieves weighted F1 scores of 0.606, 0.342, and 0.287 on 10, 24, and 30 emotion classes, respectively. Additionally, we identify challenges that must be addressed to enhance affective vocal burst recognition for human-robot interaction. Code available at /github.com/rahullabs/Sigh

This paper presents a novel method to quantify Trust in HRI. It proposes an HRI framework for estimating the Robot Trust towards the Human in the context of a narrow and specified task. The framework produces a real-time estimation of an AI agent's Artificial Trust towards a Human partner interacting with a mobile teleoperation robot. The approach for the framework is based on principles drawn from Theory of Mind, including information about the human state, action, and intent. The framework creates the ATTUNE model for Artificial Trust Towards Human Operators. The model uses metrics on the operator's state of attention, navigational intent, actions, and performance to quantify the Trust towards them. The model is tested on a pre-existing dataset that includes recordings (ROSbags) of a human trial in a simulated disaster response scenario. The performance of ATTUNE is evaluated through a qualitative and quantitative analysis. The results of the analyses provide insight into the next stages of the research and help refine the proposed approach.

Humanoid robots often struggle to express the intricate and authentic facial expressions characteristic of humans, potentially hampering user engagement. To address this challenge, we introduce a comprehensive two-stage methodology to empower our autonomous affective robot with the capacity to exhibit rich and natural facial expressions. In the initial stage, we present an innovative action unit (AU) driven facial expression disentangled synthesis method, enabling the generation of nuanced robot facial expression images guided by AUs. By harnessing facial AUs within a framework of weakly supervised learning, we effectively surmount the scarcity of paired training data (comprising source and target facial expression images). To preserve the integrity of AUs while mitigating identity interference, we leverage a latent facial attribute space to disentangle expression-related and expression-unrelated cues, employing solely the former for expression synthesis. In the subsequent phase, we actualize an affective robot endowed with multifaceted degrees of freedom for facial movements, facilitating the embodiment of the synthesized fine-grained facial expressions. We devise a specialized motor command mapping network that serves as a conduit between the generated expression images and the robot's realistic facial responses. By utilizing the physical motor positions as constraints, we refine the prediction of precise motor commands from the robot's generated facial expressions. This refinement process ensures that the robot's facial movements authentically express accurate and natural expressions. Finally, qualitative and quantitative evaluations on the benchmarking Emotionet dataset verify the effectiveness of the proposed generation method. Results on the self-developed affective robot indicate that our method achieves a promising generation of specific facial expressions with given AUs, significantly enhancing the affective human–robot interaction.

The lower extremity exoskeleton, which can sense the neural motion state of the human body and then provide motion assistance, is gradually replacing the traditional wheelchairs and assistive devices, making many patients with disabilities or movement disorders able to regain the walking function. This survey provides a comprehensive review on recent technological advances in lower extremity neurorehabilitation exoskeleton from the perspectives of sensing, gait dynamics, and human–robot collaboration. For each technology category, a detailed comparison among state-of-the-art solutions is provided. The results show that the exoskeleton has been greatly improved in mechanical and learning ability. However, some issues, such as adaptability, safety, and efficiency still restrict the development of exoskeleton technology. To address these problems, the remaining open challenges and future directions to improve intelligence, sensing, gait analysis, trust, efficiency, generalization, and power consumption of exoskeleton are also presented and discussed.

Since most existing facial expression recognition methods depend on deep learning models trained in isolation on a facial expression image corpora, once employed in scenarios that are different from those in the corpora, they usually demand ad-hoc retraining to be able to perform better in the expression recognition task for new scenarios. Furthermore, most of these facial expression recognition methods are inconsistent when recognising person-specific expressions or are incapable of adjusting to real-world scenarios where data is exclusively obtainable incrementally. In this paper, we present a face incremental expression recognition model, where we utilise domain incremental learning methods to learn individual facial features of facial expressions. We assume that each individual's facial expression (domain) is presented to the model one domain at a time. We assessed our model's ability to remember previously seen domains (individual's facial expression) and incrementally perform on new face domains. Our model improves performance compared to a non-incremental learning model and an incremental learning model in facial expression recognition for individual data with different expression classes.

Communicating shapes our social word. For a robot to be considered social and being consequently integrated in our social environment it is fundamental to understand some of the dynamics that rule human-human communication. In this work, we tackle the problem of Addressee Estimation, the ability to understand an utterance's addressee, by interpreting and exploiting non-verbal bodily cues from the speaker. We do so by implementing an hybrid deep learning model composed of convolutional layers and LSTM cells taking as input images portraying the face of the speaker and 2D vectors of the speaker's body posture. Our implementation choices were guided by the aim to develop a model that could be deployed on social robots and be efficient in ecological scenarios. We demonstrate that our model is able to solve the Addressee Estimation problem in terms of addressee localisation in space, from a robot ego-centric point of view.

Human-robot interaction (HRI) research is progressively addressing multi-party scenarios, where a robot interacts with more than one human user at the same time. Conversely, research is still at an early stage for human-robot collaboration The use of machine learning techniques to handle such type of collaboration requires data that are less feasible to produce than in a typical HRC setup. This work outlines scenarios of concurrent tasks for non-dyadic HRC applications. Based upon these concepts, this study also proposes an alternative way of gathering data regarding multi-user activity, by collecting data related to single users and merging them in post-processing, to reduce the effort involved in producing recordings of pair settings. To validate this statement, 3D skeleton poses of activity of single users were collected and merged in pairs. After this, such datapoints were used to separately train a long short-term memory (LSTM) network and a variational autoencoder (VAE) composed of spatio-temporal graph convolutional networks (STGCN) to recognise the joint activities of the pairs of people. The results showed that it is possible to make use of data collected in this way for pair HRC settings and get similar performances compared to using training data regarding groups of users recorded under the same settings, relieving from the technical difficulties involved in producing these data. The related code and collected data are publicly available11The code repository is available at: https://github.com/francescosemeraro/Multi_User_through_Single_User.git. The data repository is available at: https://figshare.com/s/64bd023e968e6eb096f3..

Social robots are becoming increasingly present in our daily lives and will continue to be integrated into society to help people with their daily routines. In this paper, we create a general motion dictionary for the NAO robot, to generate emotional gestures when the robot is interacting with humans. We implemented the motions in the context of a museum setting, wherein NAO interacts with visitors as a guide. We present a Motion Dictionary which integrates each gesture’s features and the corresponding emotions. By using the Choregraphe simulator to create the motions and validate them with a real robot, we intend to simplify and help with the generation of emotional gestures for human-robot interaction.

In order to achieve a widespread adoption of social robots in the near future, we need to design intelligent systems that are able to autonomously understand our beliefs and preferences. This will pave the foundation for a new generation of robots able to navigate the complexities of human societies. To reach this goal, we look into Theory of Mind (ToM): the cognitive ability to understand other agents’ mental states. In this paper, we rely on a probabilistic ToM model to detect when a human has false beliefs with the purpose of driving the decision-making process of a collaborative robot. In particular, we recreate an established psychology experiment involving the search for a toy that can be secretly displaced by a malicious individual. The results that we have obtained in simulated experiments show that the agent is able to predict human mental states and detect when false beliefs have arisen. We then explored the set-up in a real-world human interaction to assess the feasibility of such an experiment with a humanoid social robot.

The number of contexts in which a word is encountered, or contextual diversity, has been shown to be a relevant predictor of word-naming and lexical decision times. In this work we present an end-to-end scenario in which we collect data with a humanoid robot in three different contextual diversity levels, use the data to train a cognitive architecture with the objective of mirroring the same phenomenon observed in the literature, and ultimately we test the model by collecting test data with the robot and matching them with the learned word-object mappings. Results show that the approach manages to capture and describe successfully a computational representation of the impact of contextual diversity on word-object mapping, showing how with greater contextual diversity the mapping is more precise compared to the cases with lower diversity.

Trust is essential for social interactions, including those between humans and social artificial agents, such as robots. Several robot-related factors can contribute to the formation of trust. However, previous work has often treated trust as an absolute concept, whereas it is highly context-dependent, and it is possible that some robot-related features will influence trust in some contexts, but not in others. In this paper, we present the results of two video-based online studies aimed at investigating the role of robot voice and empathic behaviour on trust formation in a general context as well as in a task-specific context. We found that voice influences trust in the specific context, with no effect of voice or empathic behaviour in the general context. Thus, context mediated whether robot-related features play a role in people’s trust formation towards robots.

Learning fine-grained movements is a challenging topic in robotics, particularly in the context of robotic hands. One specific instance of this challenge is the acquisition of fingerspelling sign language in robots. In this paper, we propose an approach for learning dexterous motor imitation from video examples without additional information. To achieve this, we first build a URDF model of a robotic hand with a single actuator for each joint. We then leverage pre-trained deep vision models to extract the 3D pose of the hand from RGB videos. Next, using state-of-the-art reinforcement learning algorithms for motion imitation (namely, proximal policy optimization and soft actor-critic), we train a policy to reproduce the movement extracted from the demonstrations. We identify the optimal set of hyperparameters for imitation based on a reference motion. Finally, we demonstrate the generalizability of our approach by testing it on six different tasks, corresponding to fingerspelled letters. Our results show that our approach is able to successfully imitate these fine-grained movements without additional information, highlighting its potential for real-world applications in robotics.

The ability of a humanoid robot to imitate facial expressions with simultaneous head motions is crucial to natural human–robot interaction. This mirrored behavior from human beings to humanoid robots has high demands of similarity and real-time performance. To fulfill these needs, this article proposes a real-time robotic mirrored behavior of facial expressions and head motions based on lightweight networks. First, a humanoid robot that can change the state of its facial organs and neck through servo displacement is developed to achieve the mirrored behavior of facial expressions and head motions. Second, to overcome the high latency caused by deep learning models running in embedded devices, a lightweight deep learning network is constructed for detecting facial feature points, which can reduce model size and improve running speed without affecting the performance of the model. Finally, a mapping relationship of 68 facial feature points to optimal servo displacements is established to realize the mirrored behavior from human beings to humanoid robots. The experimental results show that the facial feature point recognition method based on the lightweight model performs better than other state-of-the-art methods, and our head motion tracking method can maintain high accuracy compared with the gold standard optical motion capture system NOKOV. Overall, our method ensures the accurate and real-time generation of robot mirrored behavior and has a certain reference value for the efficient and natural interaction between humans and robots.

For dimensional emotion recognition, electroencephalography (EEG) signals and electrooculogram (EOG) signals are often combined to improve the performance of classifiers, as each of them provides complementary features to the other. In this article, we combine the EEG signal on the relevant channels with the EOG signal to boost the recognition accuracy. We first explore the mutual information (MI) of all EEG channels and only select emotion-related channels, i.e., channels with more MI are retained, since the emotion recognition performance can be degraded by the interference between uncorrelated channels, while the computational complexity is significant if all EEG channels are used for recognition. While the optimal lengths of EEG and EOG signals for emotion recognition are still uncertain, we systematically investigate the effects of time-window size on emotion recognition. This strategy not only increases the number of training samples, but also reduces the feature redundancy. At this stage, we not only extract multiple statistical features but also employ the increment entropy to find abrupt changes in EEG signals. The experimental results show that 13 out of 32 EEG channels were selected by the proposed channel selection algorithm, and these selected channels can already produce accurate emotion predictions. We found that using optimal time-windows to split EEG and EOG signals into several thin slices and then combine them can further enhance the emotion recognition performance, where the time-windows of 4, 5, 6, and 10 s allow the combined signals to achieve very high accuracy.

To be seamlessly integrated in human-centered environments, robots are expected to have intelligent social capabilities on top of their physical abilities. To this end, research in artifi-cial intelligence and human-robot interaction face two major challenges. Firstly, robots need to cope with uncertainty during interaction, especially when dealing with factors that are not fully observable and hard to infer (latent variables) such as the states representing the dynamic environment and human behavior (e.g., intents, goals, preferences). Secondly, robots need to communicate their behaviors to agents (humans and other robots in the environment) in a clear and understandable manner. Therefore, robots need to be context-aware: being able to perceive and understand their surroundings, and adapt their functionalities accordingly.

Inspired by recent developments in natural language processing, we propose a novel approach to sign language processing based on phonological properties validated by American Sign Language users. By taking advantage of datasets composed of phonological data and people speaking sign language, we use a pretrained deep model based on mesh reconstruction to extract the 3D coordinates of the signers keypoints. Then, we train standard statistical and deep machine learning models in order to assign phonological classes to each temporal sequence of coordinates.Our paper introduces the idea of exploiting the phonological properties manually assigned by sign language users to classify videos of people performing signs by regressing a 3D mesh. We establish a new baseline for this problem based on the statistical distribution of 725 different signs. Our best-performing models achieve a micro-averaged F1-score of 58% for the major location class and 70% for the sign type using statistical and deep learning algorithms, compared to their corresponding baselines of 35% and 39%.

The ability to impute mental states to oneself or others, or Theory of Mind (ToM), has been intrinsically linked to trust between humans. However, less is known about how a robot mimicking ToM affects users’ trust and behaviour. We explore this through an online study, where we compare three robot personas in a cooperative maze navigation task: one neutral, one that explains its reasoning in technical terms, and one that mimics ToM. We show that ToM influences human decision-making behaviour and trust in a way that makes it more appropriate with respect to the competencies of the robot. This is key for human-robot collaboration and adoption of robotics moving forward.

There is a growing interest in the use of robotics and AI in the nuclear industry, however it is important to ensure these systems are ethically grounded, trustworthy and safe. An emerging technique to address these concerns is the use of explainability. In this paper we present the results of an interview study with nuclear industry experts to explore the use of explainable intelligent systems within the field. We interviewed 16 participants with varying backgrounds of expertise, and presented two potential use cases for evaluation; a navigation scenario and a task scheduling scenario. Through an inductive thematic analysis we identified the aspects of a deployment that experts want to know from explainable systems and we outline how these associate with the folk conceptual theory of explanation, a framework in which people explain behaviours. We established that an intelligent system should explain its reasons for an action, its expectations of itself, changes in the environment that impact decision making, probabilities and the elements within them, safety implications and mitigation strategies, robot health and component failures during decision making in nuclear deployments. We determine that these factors could be explained with cause, reason, and enabling factor explanations.

Wearable inertial motion capture, a new type of motion capture technology, mainly estimates the human posture in 3-D space through multisensor data fusion. The available method for sensor fusion is usually aided by magnetometers to remove the drift error in yaw angle estimation, which in turn limits their application in the presence of a complex magnetic field environment. In this article, an extended Kalman filter (EKF) data fusion method is proposed to fuse the 9-axis sensor data. Meanwhile, the heuristic drift reduction (HDR) method is used to calibrate the accumulated error of a heading angle. In addition, the position in 3-D space is estimated by the foot-mounted zero-velocity-update (ZUPT) technique. Combining 3-D attitude and position, a biomechanical model of the human body is established to track the motion of a real human body. The EKF algorithm and position estimation methods are benchmarked against the golden standard, optical motion capture system, for various indoor experiments. In addition, various outdoor experiments are also conducted to verify the reliability of the proposed method. The results show that the proposed algorithm outperforms the available attitude estimation model in motion tracking and is feasible for 3-D human motion capture.

Composing musical ideas longer than motifs or figures is still rare in music generated by machine learning methods, a problem that is commonly referred to as the lack of long-term structure in the generated sequences. In addition, the evaluation of the structural complexity of artificial compositions is still a manual task, requiring expert knowledge, time and involving subjectivity which is inherent in the perception of musical structure. Based on recent advancements in music structure analysis, we automate the evaluation process by introducing a collection of measures that can objectively describe structural properties of the music signal. This is done by segmenting music hierarchically, and computing our measures on the resulting hierarchies to characterise the decomposition process of music into its structural components. We tested our method on a dataset collecting music with different degrees of structural complexity, from random and computer-generated pieces to real compositions of different genres and formats. Results indicate that our method can discriminate between these classes of complexity and identify further non-trivial subdivisions according to their structural properties. Our work thus contribute a simple yet effective framework for the evaluation of music generation models in regard to their ability to create structurally meaningful compositions.

As artificial systems are starting to be widely deployed in real-world settings, it becomes critical to provide them with the ability to discriminate between different informants and to learn from reliable sources. Moreover, equipping an artificial agent to infer beliefs may improve the collaboration between humans and machines in several ways. In this article, we propose a hybrid cognitive architecture, called Thrive, with the purpose of unifying in a computational model recent discoveries regarding the underlying mechanism involved in trust. The model is based on biological observations that confirmed the role of the midbrain in trial-and-error learning, and on developmental studies that indicate how essential is a theory of mind in order to build empathetic trust. Thrive is build on top of an actor-critic framework that is used to stabilize the weights of two self-organizing maps. A Bayesian network embeds prior knowledge into an intrinsic environment, providing a measure of cost that is used to boostrap learning without an external reward signal. Following a developmental robotics approach, we embodied the model in the iCub humanoid robot and we replicated two psychological experiments. The results are in line with real data, and shed some light on the mechanisms involved in trust-based learning in children and robots.

Trust is a critical issue in human–robot interactions as it is at the base of the establishment of solid relationships. Theory of Mind (ToM) is the cognitive skill that allows us to understand what others think and believe. Several studies in HRI and psychology suggest that trust and ToM are interdependent concepts since we trust another agent based on our representation of its actions, beliefs, and intentions. However, very few works take ToM of the robot into consideration while studying trust in HRI. In this paper, we aim to examine whether the perception of ToM abilities on a robotic agent influences human-robot trust over time in an iterative game scenario. To this end, participants played an Investment Game with a humanoid robot (Pepper) that was presented as having either low-level ToM or high-level ToM. During the game, the participants were asked to pick a sum of money to invest in the robot. The amount invested was used as the main measurement of human-robot trust. Our experimental results show that robots possessing a high-level of ToM abilities were trusted more than the robots presented with low-level ToM skills.

With advances in the field of machine learning, service robots are envisioned to become more present. The COVID-19 pandemic has accelerated this need. One such example would be coffee shops, which have become intrinsic to our everyday lives. Yet, serving an excellent cup of coffee is not trivial as a coffee blend typically comprises rich aromas, indulgent and unique flavours. Our work addresses this by proposing a computational model which recommends optimal coffee beans resulting from users' preferences. Given coffee properties (objective features), we apply different supervised learning techniques to predict coffee qualities (subjective features). We then consider an unsupervised learning method to analyse the relationship between coffee beans in the subjective feature space. Evaluated on a real coffee beans dataset based on digitised reviews, our results illustrate that the proposed computational model gives up to 92.7 percent recommendation accuracy for coffee prediction. From this, we propose how it can be deployed on a robot.

Theories on social learning indicate that imitative choices are usually performed whenever copying the others' behaviour has no additional cost. Here, we extended such investigations of social learning to Human-Robot Interaction (HRI). Participants played the Economic Investment Game with a robot banker while observing another robot player also investing in the robot banker. By manipulating the robot banker payoff, three conditions of unfairness were created: (1) unfair payoff for the participants, (2) unfair payoff for the robot player and (3) unfair payoff for both. Results showed that when the payoff was low for the participants and high for the robot player, participants invested more money in the robot banker than when both parties received a low return. Also, for this specific condition, participants' investments increased further with a more interactive robot player (defined as demonstrating increased attention, congruent movements and speech) This suggests that social and cognitive human competencies can be used and transposed to non-human agents. Further, imitation can potentially be extended to HRI, with interactivity likely having a key role in increasing this effect.

Recurrent Neural Network (RNN) have been used for sequence-related learning tasks, such as language and action, in the field of cognitive robotics. Gated mechanisms used in LSTM and GRU perform well in remembering long-term dependency. But to better mimic the neural dynamics in cognitive processes, the Multiple Time-scales (MT) RNN uses a hierarchical organization of memory updates which is similar to human cognition. Since the MT feature is typically used with a vanilla RNN or different gated mechanisms, its effect on the updates and training is still not fully uncovered. Therefore, we conduct a comparative experiment on two MT recurrent neural network models, i.e. the Multiple Time-Scale Recurrent Neural Network (MTRNN) and the Multiple Time-Scale Gated Recurrent Unit (MTGRU), for action sequence learning in robotics. The experiment shows that the MTRNN model can be used in learning tasks with low requirements for learning of long-term dependency due to its low computation. On the other hand, the MTGRU model is appropriate for learning the longterm dependency. Furthermore, because of the duplicated feature of the MT and the GRU feature, we also propose a simplified MTGRU model, named Multiple Time-scale SingleGate Recurrent Unit (MTSRU) which could reduce computational cost while it achieves the similar performance as the original version.

Trust is a critical issue in human-robot interactions (HRI) as it is the core of human desire to accept and use a non-human agent. Theory of Mind (ToM) has been defined as the ability to understand the beliefs and intentions of others that may differ from one's own. Evidences in psychology and HRI suggest that trust and ToM are interconnected and interdependent concepts, as the decision to trust another agent must depend on our own representation of this entity's actions, beliefs and intentions. However, very few works take ToM of the robot into consideration while studying trust in HRI. In this paper, we investigated whether the exposure to the ToM abilities of a robot could affect humans' trust towards the robot. To this end, participants played a Price Game with a humanoid robot (Pepper) that was presented having either low-level ToM or high-level ToM. Specifically, the participants were asked to accept the price evaluations on common objects presented by the robot. The willingness of the participants to change their own price judgement of the objects (i.e., accept the price the robot suggested) was used as the main measurement of the trust towards the robot. Our experimental results showed that robots possessing a high-level of ToM abilities were trusted more than the robots presented with low-level ToM skills.

Social interaction is the new frontier in contemporary robotics: we want to build robots that blend with ease into our daily social environments, following their norms and rules. The cognitive skill that bootstraps social awareness in humans is known as "intention reading" and it allows us to interpret other agents' actions and assign them meaning. Given its centrality for humans, it is likely that intention reading will foster the development of robotic social understanding. In this paper, we present an artificial cognitive architecture for intention reading in human-robot interaction (HRI) that makes use of social cues to disambiguate goals. This is accomplished by performing a low-level action encoding paired with a high-level probabilistic goal inference. We introduce a new clustering algorithm that has been developed to differentiate multi-sensory human social cues by performing several levels of clustering on different feature-spaces, paired with a Bayesian network that infers the underlying intention. The model has been validated through an interactive HRI experiment involving a joint manipulation game performed by a human and a robotic arm in a toy block scenario. The results show that the artificial agent was capable of reading the intention of its partner and cooperate in mutual interaction, thus validating the novel methodology and the use of social cues to disambiguate goals, other than demonstrating the advantages of intention reading in social HRI.

With the global population aging at an alarming rate, the need to find alternative ways to deliver quality assistance is becoming a pressing concern for health and care systems. To promptly provide companion-like assistance, robots need to gain social intelligence in an autonomous way, without relying on human operators. The work described in this paper aims to develop a deep learning agent that, by means of convolutional neural network architecture in the decision making loop, could understand when and how, to interact with one, or more people, gathered in a room. This was done by training a robot to assess the level of user engagement at the initiation of the interaction, so that the robot could detect the person most willing to start interacting. The robot's performance as a deep learning agent was tested through an experiment with potential ''end-users'', following an iterative process, over four days. The deep learning agent was able to take the right decision 59% of the times by the end of the experiment, from an initial success rate of 44% on the first day, proving the potential of such technologies in this application field.

In the early stages of infant development, gestures and speech are integrated during language acquisition. Such a natural combination is therefore a desirable, yet challenging, goal for fluid human-robot interaction. To achieve this, we propose a multimodal deep learning architecture, for comprehension of complementary gesture-word combinations, implemented on an iCub humanoid robot. This enables human-assisted language learning, with interactions like pointing at a cup and labelling it with a vocal utterance. We evaluate various depths of the Mask Regional Convolutional Neural Network (for object and wrist detection) and the Residual Network (for gesture classification). Validation is carried out with two deictic gestures across ten real-world objects on frames recorded directly from the iCub's cameras. Results further strengthen the potential of gesture-word combinations for robot language acquisition.

In this paper we present a neural architecture to learn a bi-directional mapping between actions and language. We implement a Multiple Timescale Long Short-Term Memory (MT-LSTM) network comprised of 7 layers with different timescale factors, to connect actions to language without explicitly learning an intermediate representation. Instead, the model self-organizes such representations at the level of a slow-varying latent layer, linking action branch and language branch at the center. We train the model in a bi-directional way, learning how to produce a sentence from a certain action sequence input and, simultaneously, how to generate an action sequence given a sentence as input. Furthermore we show this model preserves some of the generalization behaviour of Multiple Timescale Recurrent Neural Networks (MTRNN) in generating sentences and actions that were not explicitly trained. We compare this model with a number of different baseline models, confirming the importance of both the bi-directional training and the multiple timescales architecture. Finally, the network was evaluated on motor actions performed by an iCub robot and their corresponding letter-based description. The results of these experiments are presented at the end of the paper.

The employment of Social Assistive Robots (SARs) for monitoring elderly users represents a valuable gateway for at-home assistance. Their deployment in the house of the users can provide effective opportunities for early detection of Mild Cognitive Impairment (MCI), a condition of increasing impact in our aging society, by means of digitalized cognitive tests. In this work, we present a system where a specific set of cognitive tests is selected, digitalized, and integrated with a robotic assistant, whose task is the guidance and supervision of the users during the completion of such tests. The system is then evaluated by means of an experimental study involving potential future users, in order to assess its acceptability and identify key directions for technical improvements.

Deep reinforcement learning has proven to be a great success in allowing agents to learn complex tasks. However, its application to actual robots can be prohibitively expensive. Furthermore, the unpredictability of human behavior in human-robot interaction tasks can hinder convergence to a good policy. In this paper, we present an architecture that allows agents to learn models of stochastic environments and use them to accelerate learning. We descirbe how an environment model can be learned online and used to generate synthetic transitions, as well as how an agent can leverage these synthetic data to accelerate learning. We validate our approach using an experiment in which a robotic arm has to complete a task composed of a series of actions based on human gestures. Results show that our approach leads to significantly faster learning, requiring much less interaction with the environment. Furthermore, we demonstrate how learned models can be used by a robot to produce optimal plans in real world applications.

The ageing of the population deeply impacts on the social costs relative to health care. The use of modern technologies is one of the most promising approaches, under current study, to reduce such impact. In this demonstration, we propose a framework that can be employed for at-home assessment of Mild Cognitive Impairment (MCI). It is composed by a set of digitalized cognitive tests, developed from their paper-and-pencil counterparts, and by a Virtual Caregiver, which oversees the test execution and provides instructions.

The predictive processing (PP) hypothesizes that the predictive inference of our sensorimotor system is encoded implicitly in the regularities between perception and action. We propose a neural architecture in which such regularities of active inference are encoded hierarchically. We further suggest that this encoding emerges during the embodied learning process when the appropriate action is selected to minimize the prediction error in perception. Therefore, this predictive stream in the sensorimotor loop is generated in a top-down manner. Specifically, it is constantly modulated by the motor actions and is updated by the bottom-up prediction error signals. In this way, the top-down prediction originally comes from the prior experience from both perception and action representing the higher levels of this hierarchical cognition. In our proposed embodied model, we extend the PredNet Network, a hierarchical predictive coding network, with the motor action units implemented by a multi-layer perceptron network (MLP) to modulate the network top-down prediction. Two experiments, a minimalistic world experiment, and a mobile robot experiment are conducted to evaluate the proposed model in a qualitative way. In the neural representation, it can be observed that the causal inference of predictive percept from motor actions can be also observed while the agent is interacting with the environment.

How can high dimensional robots learn general sets of skills from experience in the real world? Many previous approaches focus on maximizing a single utility function and require large datasets of experience to do this, something that is not possible to collect outside of simulation as every data point is expensive both in time and in a potential wear down of the robot. This paper addresses this question using a newly developed framework called Finite Element Goal Babbling (FEGB). FEGB is an online learning method that aims at providing general control over some measurable feature, in contrast to optimizing it to some given utility function. It generalizes standard goal babbling by breaking down the full learning problem into local sub-problems, and combining it with a planner that learns how to navigate between these subproblems. We test FEGB using a real humanoid robot Nao, and find that it could quickly learn to robustly control its body orientation. After only 20-30 minutes of training, the robot could freely move into any body orientation between lying on either side and on its back. Rapid learning of body orientation control in high dimensional real robots is largely an unexplored field of robotics, and although many challenges remain, FEGB shows a feasible approach to the problem.

The world population is ageing at a dramatic rate, raising new challenges for social and health care systems. Sometimes, assistance can simply derive from a social interaction between a robotic platform and human users. In these cases, robots cannot rely on human operators. Therefore, they need to gain social intelligence in a fully autonomous way. The focus of this paper is on the initial steps needed to implement a completely autonomous robotic agent able to adapt itself to its users. For this reason, an interactive data collection was carried out to gather a dataset from which the robot could learn how to respond to its users in different situations. From these data, a first evaluation of the performances of the deep learning agent, embodied in the robot, has been completed. The agent was able to generalize to new sets of test data. The study explored how, using modern machine learning algorithms, a robot could learn to understand if, and how, to interact with one, or more people, gathered in a room. This was done by training a robot to read the level of the engagement of the users at the initiation of the interaction.

ANNABELL is a cognitive system entirely based on a large-scale neural architecture capable of learning to communicate through natural language starting from a tabula rasa condition. In order to shed light on the level of cognitive development required for language acquisition, in this work the model is used to study the acquisition of a new language, namely Albanian, in addition to English. The aim is to evaluate in a completely different and more complex language the ability of the model to acquire new information through several examples introduced in the new language and to process the acquired information, answering questions that require the use of different language patterns. The results show that the system is capable of learning cumulatively in either language and to develop a broad range of language processing functionalities in both languages.

research-article Share on ACM Transactions on Interactive Intelligent Systems (TiiS) Special Issue on Trust and Influence in Intelligent Human-Machine Interaction Authors: Benjamin A. Knott The Office of Naval Research Global, Roppongi, Tokyo, Japan The Office of Naval Research Global, Roppongi, Tokyo, JapanView Profile , Jonathan Gratch University of Southern California, USA University of Southern California, USAView Profile , Angelo Cangelosi Plymouth University, USA Plymouth University, USAView Profile , James Caverlee Texas A8M University, USA Texas A8M University, USAView Profile Authors Info & Claims ACM Transactions on Interactive Intelligent SystemsVolume 8Issue 4December 2018 Article No.: 25pp 1–3https://doi.org/10.1145/3281451Published:16 November 2018Publication History 0citation395DownloadsMetricsTotal Citations0Total Downloads395Last 12 Months62Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We present here a learning system using the iCub humanoid robot and the SpiNNaker neuromorphic chip to solve the real-world task of object-specific attention. Integrating spiking neural networks with robots introduces considerable complexity for questionable benefit if the objective is simply task performance. But, we suggest, in a cognitive robotics context, where the goal is understanding how to compute, such an approach may yield useful insights to neural architecture as well as learned behavior, especially if dedicated neural hardware is available. Recent advances in cognitive robotics and neuromorphic processing now make such systems possible. Using a scalable, structured, modular approach, we build a spiking neural network where the effects and impact of learning can be predicted and tested, and the network can be scaled or extended to new tasks automatically. We introduce several enhancements to a basic network and show how they can be used to direct performance toward behaviorally relevant goals. Results show that using a simple classical spike-timing-dependent plasticity (STDP) rule on selected connections, we can get the robot (and network) to progress from poor task-specific performance to good performance. Behaviorally relevant STDP appears to contribute strongly to positive learning: "do this" but less to negative learning: "don't do that." In addition, we observe that the effect of structural enhancements tends to be cumulative. The overall system suggests that it is by being able to exploit combinations of effects, rather than any one effect or property in isolation, that spiking networks can achieve compelling, task-relevant behavior.

Group emotion recognition in the wild is a challenging problem, due to the unstructured environments in which everyday life pictures are taken. Some of the obstacles for an effective classification are occlusions, variable lighting conditions, and image quality. In this work we present a solution based on a novel combination of deep neural networks and Bayesian classifiers. The neural network works on a bottom-up approach, analyzing emotions expressed by isolated faces. The Bayesian classifier estimates a global emotion integrating top-down features obtained through a scene descriptor. In order to validate the system we tested the framework on the dataset released for the Emotion Recognition in the Wild Challenge 2017. Our method achieved an accuracy of 64.68% on the test set, significantly outperforming the 53.62% competition baseline.

Two different configurations of Liquid State Machine (LSM), a special type of Reservoir Computing with internal nodes modelled as spiking neurons, implementing multiple columns (Modular and Monolithic approaches) are tested against the decimation of neurons, connections and entire columns in order to verify which one can better withstand the damage. Based on the neurorobotics outlook, this work is part of a bigger project that aims to apply artificial neural networks to the control of humanoid robots. Therefore, as a benchmark, we made use of a robotic task where an LSM is trained to generate the joint angles needed to command a simulated version of the collaborative robot BAXTER to draw a square on top of a table. The final drawn shape is analysed through Dynamical Time Warping to generate a cost value based on how close the produced drawing is to the original shape. Our results show both approaches, Modular and Monolithic, had a similar behaviour, however the Modular was better at withstanding the decimation of neurons when it was concentrated in a single column.

Biological neural networks are able to control limbs in different scenarios, with high precision and robustness. As neural networks in living beings communicate through spikes, modern neuromorphic systems try to mimic them making use of spike-based neuron models. Liquid State Machines (LSM), a special type of Reservoir Computing system made of spiking units, when it was first introduced, had plasticity on an external layer and also through Short-Term Plasticity (STP) within the reservoir itself. However, most neuromorphic hardware currently available does not implement both Short-Term Depression and Facilitation and some of them don't support STP at all. In this work, we test the impact of STP in an experimental way using a 2 degrees of freedom simulated robotic arm controlled by an LSM. Four trajectories are learned and their reproduction analysed with Dynamic Time Warping accumulated cost as the benchmark. The results from two different set-ups showed the use of STP in the reservoir was useful for one out of three tested trajectories, though not computationally cost-effective for this particular robotic task.

The abstraction tasks are challenging for multi-modal sequences as they require a deeper semantic understanding and a novel text generation for the data. Although the recurrent neural networks (RNN) can be used to model the context of the time-sequences, in most cases the long-term dependencies of multi-modal data make the back-propagation through time training of RNN tend to vanish in the time domain. Recently, inspired from Multiple Time-scale Recurrent Neural Network (MTRNN) [1], an extension of Gated Recurrent Unit (GRU), called Multiple Time-scale Gated Recurrent Unit (MTGRU), has been proposed [2] to learn the long-term dependencies in natural language processing. Particularly it is also able to accomplish the abstraction task for paragraphs given that the time constants are well defined. In this paper, we compare the MTRNN and MTGRU in terms of its learning performances as well as their abstraction representation on higher level (with a slower neural activation). This was done by conducting two studies based on a smaller dataset (two-dimension time sequences from non-linear functions) and a relatively large data-set (43-dimension time sequences from iCub manipulation tasks with multi-modal data). We conclude that gated recurrent mechanisms may be necessary for learning long-term dependencies in large dimension multi-modal data-sets (e.g. learning of robot manipulation), even when natural language commands was not involved. But for smaller learning tasks with simple time-sequences, generic version of recurrent models, such as MTRNN, were sufficient to accomplish the abstraction task.

We investigated the perceived credibility of statements made by robots, hypothesising that people are more likely to believe robots with humanlike characteristics than those that are less anthropomorphic. We also examined whether prior experience with a humanlike robot would lead people to extend this advantage to the less-anthropomorphic robot. A measure of credibility was provided by agreement on the pricing of objects, where participants negotiated with either a more (iCub) or less-anthropomorphic robot (Scitos G5) that was engaged in more (using social gaze) or less-humanlike (fixed gaze) social behaviour. In the first experiment participants only interacted with Scitos G5, in the second they interacted with Scitos G5 only after having first interacted iCub. Results showed that iCub was more credible than Scitos G5, and was the only robot to benefit from the use of social gaze. It was also found that the credibility of the Scitos G5 was higher after participants were `primed' by prior exposure to the iCub.

This paper considers the issue of fully automatic emotion classification on 2D faces. In spite of the great effort done in recent years, traditional machine learning approaches based on hand-crafted feature extraction followed by the classification stage failed to develop a real-time automatic facial expression recognition system. The proposed architecture uses Convolutional Neural Networks (CNN), which are built as a collection of interconnected processing elements to simulate the brain of human beings. The basic idea of CNNs is to learn a hierarchical representation of the input data, which results in a better classification performance. In this work we present a block-based CNN algorithm, which uses noise, as data augmentation technique, and builds batches with a balanced number of samples per class. The proposed architecture is a very simple yet powerful CNN, which can yield state-of-the-art accuracy on the very competitive benchmark algorithm of the Extended Cohn Kanade database.

How exactly our brain works is still an open question, but one thing seems to be clear: biological neural systems are computationally powerful, robust and noisy. Using the Reservoir Computing paradigm based on Spiking Neural Networks, also known as Liquid State Machines, we present results from a novel approach where diverse and noisy parallel reservoirs, totalling 3,000 modelled neurons, work together receiving the same averaged feedback. Inspired by the ideas of action learning and embodiment we use the safe and flexible industrial robot BAXTER in our experiments. The robot was taught to draw three different 2D shapes on top of a desk using a total of four joints. Together with the parallel approach, the same basic system was implemented in a serial way to compare it with our new method. The results show our parallel approach enables BAXTER to produce the trajectories to draw the learned shapes more accurately than the traditional serial one.

Mental rotation, a classic experimental paradigm of cognitive psychology, tests the capacity of humans to mentally rotate a seen object to decide if it matches a target object. In recent years, mental rotation has been investigated with brain imaging techniques to identify the brain areas involved. Mental rotation has also been investigated through the development of neural-network models, used to identify the specific mechanisms that underlie its process, and with neurorobotics models to investigate its embodied nature. Current models, however, have limited capacities to relate to neuro-scientific evidence, to generalise mental rotation to new objects, to suitably represent decision making mechanisms, and to allow the study of the effects of overt gestures on mental rotation. The work presented in this study overcomes these limitations by proposing a novel neurorobotic model that has a macro-architecture constrained by knowledge held on brain, encompasses a rather general mental rotation mechanism, and incorporates a biologically plausible decision making mechanism. The model was tested using the humanoid robot iCub in tasks requiring the robot to mentally rotate 2D geometrical images appearing on a computer screen. The results show that the robot gained an enhanced capacity to generalise mental rotation to new objects and to express the possible effects of overt movements of the wrist on mental rotation. The model also represents a further step in the identification of the embodied neural mechanisms that may underlie mental rotation in humans and might also give hints to enhance robots' planning capabilities.

Thanks to recent technological advances and the increasing interest towards the Cognitive Developmental Robotics (CDR) paradigm, many popular platforms for scientific research have been designed in order to resemble the shape of the human body. The motivation behind this strongly humanoid design is the embodied cognition hypothesis, which affirms that all aspects of cognition are shaped by aspects of the body. Thus CDR is based on a synthetic approach that aims to provide new understanding on how human beings develop their higher cognitive functions. Following this paradigm we have developed an artificial model, based on artificial neural networks, to explore finger counting and the association of number words (or tags) to the fingers, as bootstrapping for the representation of numbers in the humanoid robot iCub. In this paper, we detail experiments of our model with the iCub robotic platform. Results of the number learning with propri-oceptive data from the real platform are reported and compared with the ones obtained instead, with the simulated platform. These results support the thesis that learning the number words in sequence, along with finger configurations helps the building of the initial representation of number in the robot. Moreover, the comparison between the real and simulated iCub gives insights on the use of these platforms as a tool for CDR.

Neural networks have been used in many different robot motor-control experiments, however, so far the complexity of these neuro-controllers have remained at the similar level. The focus of this paper is to demonstrate that it is possible to scale-up these neuro-robotic controllers with GPUs leading to richer, more realistic and more complex motor control.

The work described here explores a neural network architecture that can be embedded directly in the realtime sensorimotor coordination loop of a developmental robot platform. We take inspiration from the way children are able to learn while interacting with a teacher, in particular the use of prediction of the teacher actions to improve own learning. The architecture is based on two neural networks that operate online, and in parallel, one for learning and one for prediction. A Hebbian learning rule is used to associate the high-dimensional afferent sensor input at different time-delays with the current efferent motor commands corresponding to the teacher demonstration. The predictions of future motor commands are used to limit the growth of the neural network weights, and to enable the robot to smoothly continue movements the teacher has begun. Results on a simulated iCub robot learning object interaction tasks are presented, including an analysis of the sensitivity to changes in the task setup. We also outline the first implementation on the real iCub platform.

In this paper we present the design and technical implementation of a web based Multi-Modal User Interface (MMUI) tailored for elderly users of the robotic services developed by the EU FP7 Large-Scale Integration Project Robot-Era. The project partners are working to significantly enhance the performance and acceptability of technological services for ageing well by delivering a fully realized system based on the cooperation of multiple heterogeneous robots and with the support of an Ambient Assisted Living environment. To this end, elderly users were involved in the definition of the services and in the design of the hardware and software of the robotic platforms from the first stages of the development process and in real experimentation in two test sites. In particular, here we detail the interface software system for multi-modal elderly-robot interaction. The MMUI is designed to run on any device including touch-screen mobiles and tablets that are preferred by the elderly. This is obtained by integrating web based solutions with the Robot-Era middlewares and planner. Finally we present some preliminary results of ongoing experiments to show the successful evaluation of usability by potential users and to discuss the future directions to improve the proposed MMUI software system.

This paper presents a cognitive learning system for robot recognition and composite action learning. The cognitive system of the robot is an artificial neural network trained to recognize and handle objects through imitation and back-propagation algorithm learning. The robot is first trained to learn the representation of action words, object categories and grounded language understanding. Following a human tutor's linguistic instructions, the robot autonomously transfers the grounding form directly basics knowledge to new higher level composite knowledge.

Understanding the process behind the human ability of creating mental images of events and experiences is a still crucial issue for psychologists. Mental imagery may be considered a multimodal biological simulation that activates the same, or very similar, sensorial and motor modalities that are activated when we interact with the environment in real time. Neuro-psychological studies show that neural mechanisms underlying real-time visual perception and mental visualization are the same when a task is mentally recalled. Nevertheless, the neural mechanisms involved in the active elaboration of mental images might be different from those involved in passive elaborations. The enhancement of this active and creative imagery is the aim of most psychological and educational processes, although, more empirical effort is needed in order to understand the mechanisms and the role of active mental imagery in human cognition. In this work we present some results of on ongoing investigation about mental imagery using cognitive robotics. Here we focus on the capability to estimate, from proprioceptive and visual information, the position into a soccer field when the robot acquires the goal. Results of simulation with the iCub platform are given to show that the computational model is able to efficiently estimate the robot's position. The final objective of our work is to replicate with a cognitive robotics model the mental imagery when it is used during the training phase of athletes that are allowed to imaginary practice to score a goal.

This paper presents a novel open-source software application, Aquila, developed as a part of the ITALK and RobotDoC projects. The software provides many different tools and biologically-inspired models, useful for cognitive and developmental robotics research. Aquila addresses the need for high-performance robot control by adopting the latest parallel processing paradigm, based on the NVidia CUDA technology. The software philosophy, implementation, functionalities and performance are described together with three practical examples of selected modules.

In this paper, a model based on Artificial Neural Networks (ANNs) extends the symbol grounding mechanism to abstract words for cognitive robots. The aim of this work is to obtain a semantic representation of abstract concepts through the grounding in sensorimotor experiences for a humanoid robotic platform. Simulation experiments have been developed on a software environment for the iCub robot. Words that express general actions with a sensorimotor component are first taught to the simulated robot. During the training stage the robot first learns to perform a set of basic action primitives through the mechanism of direct grounding. Subsequently, the grounding of action primitives, acquired via direct sensorimotor experience, is transferred to higher-order words via linguistic descriptions. The idea is that by combining words grounded in sensorimotor experience the simulated robot can acquire more abstract concepts. The experiments aim to teach the robot the meaning of abstract words by making it experience sensorimotor actions. The iCub humanoid robot will be used for testing experiments on a real robotic architecture.

The success of swarm behaviors often depends on the range at which robots can communicate and the speed at which they change their behavior. Challenges arise when the communication range is too small with respect to the dynamics of the robot, preventing interactions from lasting long enough to achieve coherent swarming. To alleviate this dependency, most swarm experiments done in laboratory environments rely on communication hardware that is relatively long range and wheeled robotic platforms that have omnidirectional motion. Instead, we focus on deploying a swarm of small fixed-wing flying robots. Such platforms have limited payload, resulting in the use of short-range communication hardware. Furthermore, they are required to maintain forward motion to avoid stalling and typically adopt low turn rates because of physical or energy constraints. The tradeoff between communication range and flight dynamics is exhaustively studied in simulation in the scope of Reynolds flocking and demonstrated with up to 10 robots in outdoor experiments.

Autonomous navigation and robust obstacle avoidance are prerequisites for the successful operation of a planetary rover. Typical approaches to tackling this problem rely on complex and computationally expensive navigation strategies based upon the creation of 3D maps of the environment. In contrast, this research proposes a simple artificial neural network relying on infrared sensory input as the control structure. This paper presents a unified framework for designing such control structures for a simulated rover, taking advantage of code parallelisation and the latest advances in global optimisation research. In particular, it details a 3D physics-based simulation of a planetary rover and a tool set for performing the optimisation of ANN parameters within the island model. This paper also presents preliminary results showing that the aforementioned framework can parallelise the controller design process without any loss in performance over traditional methods, and will outline research directions, which aim to take full advantage of this technique's potential.

The work presented herein describes an application of Evolutionary Robotics controller design methodologies to the domain of Micro-unmanned Aerial Vehicles (MAVs). The aim of this paper is to extend and validate preliminary results obtained through a simplified 2D simulator, to a more realistic 3D model. After a technical introduction of the newly developed simulation model, the results generated by three different experimental setups - all of them focused on autonomous navigation toward a specific target area - are described. The first scenario simply involves a single MAV navigating through a plain environment toward a non-movable target. In the second setup the target is able to move away, at different speeds, when approached by the aircraft. Finally, in the third scenario, teams consisting of more than one MAV are employed; the team members have to coordinate among themselves - exploiting implicit communication strategies - in order to reach the target at the same time. The nature of the tasks studied requires a high level of accuracy by the controllers, something which is not common in most of the ER literature.

In this paper we present a model which allows to co-evolve the morphology and the control system of realistically simulated robots (creatures). The method proposed is based on an artificial ontogenetic process in which the genotype does not specify directly the characteristics of the creatures but rather the growing rules that determine how an initial artificial embryo will develop on a fully formed individual. More specifically, the creatures are generated through a developmental process which occurs in time and space and which is realized through the progressive addition of both structural parts and regulatory substances which affect the successive course of the morphogenetic process. The creatures are provided with a distributed control system made up of several independent neural controllers embedded in the different body parts which only have access to local sensory information and which coordinate through the effects of physical actions mediated by the external environment through the emission/detection of signals which diffuse locally in space. The analysis of evolved creatures shows how they display effective morphology and control mechanisms which allow them to walk effectively and robustly both on regular and irregular terrains in all the replications of the experiment. Moreover, the obtained results show how the possibility to develop such skills can be improved by also selecting individuals on the basis of a task-independent component which reward them for the ability to coordinate the movements of their parts.

The paper presents an evolutionary robotics model of the Rover Mars robot. This work has the objective to investigate the possibility of using an alternative sensor system, based on infrared sensors, for future rovers capable of performing autonomous tasks in challenging planetary terrain environments. The simulation model of the robot and of Mars terrain is based on a physics engine. The robot control system consists of an artificial neural network trained using evolutionary computation techniques. An adaptive threshold on the infrared sensors has been evolved together with the neural control system to allow the robot to adapt itself to many different environmental conditions. The properties of the behavior obtained after the evolutionary process has been tested by measuring the generalization performance of the rover under various terrain conditions and especially under rough terrain conditions. In addition, the dynamics of the co-evolution between the controller and the threshold has been analyzed. Those analyses show that different pathways have been explored by the evolutionary process in order to adapt the sensing abilities and the control system.

Cross-situational learning is based on the idea that a learner can determine the meaning of a word by finding something in common across all observed uses of that word. Although cross-situational learning is usually modeled through stochastic guessing games in which the input data vary erratically with time (or rounds of the game), here we investigate the possibility of applying the deterministic neural modeling fields (NMF) categorization mechanism to infer the correct object-word mapping. Two different representations of the input data were considered. The first is termed object-word representation because it takes as inputs all possible object-word pairs and weighs them by their frequencies of occurrence in the stochastic guessing game. A re-interpretation of the problem within the perspective of learning with noise indicates that the cross-situational scenario produces a too low signal-to-noise ratio, explaining thus the failure of NMF to infer the correct object-word mapping. The second representation, termed context-word, takes as inputs all the objects that are in the pupil's visual field (context) when a word is uttered by the teacher. In this case we show that use of two levels of hierarchy of NMF allows the inference of the correct object-word mapping.

The work presented here focuses on the use of embodied neural network controllers for MAV (micro-unmanned aerial vehicles) teams. The computer model we have built aims to demonstrate how autonomous controllers for groups of flying robots can be successfully developed through simulations based on multi-agent systems and evolutionary robotics methodologies. We first introduce the field of autonomous flying robots, reviewing the most relevant contributes on this research field and highlighting the elements of novelty contained in our approach. We then describe the simulation model we have elaborated and the results obtained in different experimental scenarios. In all experiments, MAV teams made by four agents have to navigate autonomously through an unknown environment, reach a certain target and finally neutralize it through a self-detonation. The different setups comprise an environment with various obstacles (skyscrapers) and a fixed target, one with a moving target, and one where the target (fixed or moving) needs to be attacked cooperatively in order to be neutralized. The results obtained show how the evolved controllers are able to perform the various tasks with an accuracy level between 72% and 94% when the target has to be approached individually. The performance slightly decreases only when the target is both able to move and can only be neutralized through a coordinated operation. The paper ends with a discussion on the possible applications of autonomous MAV teams to real life scenarios.

Neural networks have been proposed as an ideal cognitive modeling methodology to deal with the symbol grounding problem. More recently, such neural network approaches have been incorporated in studies based on cognitive agents and robots. In this paper we present a new model of symbol grounding transfer in cognitive robots. Language learning simulations demonstrate that robots are able to acquire new action concepts via linguistic instructions. This is achieved by autonomously transferring the grounding from directly grounded action names to new higher-order composite actions. The robot's neural network controller permits such a grounding transfer. The implications for such a modeling approach in cognitive science and autonomous robotics are discussed.

Studies of the emergence of language focus on the evolutionary and developmental factors that affect the acquisition and auto-organization of a linguistic communication system (MacWhinney 1999, Kni...

This paper reports new simulations on an extended neural network model for the transfer of symbol grounding. It uses a hybrid and modular connectionist model, consisting of an unsupervised, self-organizing map for stimulus classification and a supervised network for category acquisition and naming. The model is based on a psychologically-plausible view of symbolic communication, where unsupervised concept formation precedes the supervised acquisition of category names. The simulation results demonstrate that grounding is transferred from symbols denoting object properties to newly acquired symbols denoting the object as a whole. The implications for cognitive models integrating neural networks and multi-agent systems are discussed.

Connection Science is seeking submissions for a special issue entitled ‘The Emergence of Language: Neural and Adaptive Agent Models’. Studies of the emergence of language focus on the evolutionary ...

The Baldwin effect has been explicitly used by Pinker and Bloom as an explanation of the origins of language and the evolution of a language acquisition device. This article presents new simulations of an artificial life model for the evolution of compositional languages. It specifically addresses the role of cultural variation and of learning costs in the Baldwin effect for the evolution of language. Results show that when a high cost is associated with language learning, agents gradually assimilate in their genome some explicit features (e.g., lexical properties) of the specific language they are exposed to. When the structure of the language is allowed to vary through cultural transmission, Baldwinian processes cause, instead, the assimilation of a predisposition to learn, rather than any structural properties associated with a specific language. The analysis of the mechanisms underlying such a predisposition in terms of categorical perception supports Deacon's hypothesis regarding the Baldwinian inheritance of general underlying cognitive capabilities that serve language acquisition. This is in opposition to the thesis that argues for assimilation of structural properties needed for the specification of a full-blown language acquisition device.

This paper describes different types of models for the evolution of communication and language. It uses the distinction between signals, symbols, and words for the analysis of evolutionary models of language. In particular, it shows how evolutionary computation techniques such as artificial life can be used to study the emergence of syntax and symbols from simple communication signals. Initially, a computational model that evolves repertoires of isolated signal is presented. This study has simulated the emergence of signals for naming foods in a population of foragers. This type of model studies communication systems based on simple signal-object associations. Subsequently, models that study the emergence of grounded symbols are discussed in general, including a detailed description of a work on the evolution of simple syntactic rules. This model focuses on the emergence of symbol-symbol relationships in evolved languages. Finally, computational models of syntax acquisition and evolution are discussed. These different types of computational models provide an operational definition of the signal/symbol/word distinction. The simulation and analysis of these types of models will help to understand the role of symbols and symbol acquisition in the origin of language.

A bstract

This paper uses a model of neural network and genetic algorithms to simulate the evolution of communication in populations of evolving neural networks. It focuses on the emergence of simple forms of syntax, i.e., the combination of two symbols. The simulation task resembles Savage-Rumbaugh and Rumbaugh's experiment (1978) on ape language and symbol acquisition. The simulation results show the evolution and cultural transmission of languages based on combination of grounded symbols. The model is analyzed according to the issues of the symbol grounding and symbol acquisition problems.

The evolution of language implies the parallel evolution of an ability to respond appropriately to signals (language understanding) and an ability to produce the appropriate signals in the appropriate circumstances (language production). When linguistic signals are produced to inform other individuals, individuals that respond appropriately to these signals may increase their reproductive chances but it is less clear what the reproductive advantage is for the language producers. We present simulations in which populations of neural networks living in an environment evolve a simple language with an informative function. Signals are produced to help other individuals categorize edible and poisonous mushrooms, in order to decide whether to approach or avoid encountered mushrooms. Language production, while not under direct evolutionary pressure, evolves as a byproduct of the independently evolving perceptual ability to categorize mushrooms.