We study hybrid execution in multi-agent reinforcement learning (MARL), a paradigm where agents aim to complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information.

We study the convergence of Q -learning-based methods in convergently non-stationary environments, particularly in the context of hierarchical reinforcement learning and of dynamic features encountered in deep reinforcement learning. We demonstrate that Q -learning achieves convergence in tabular representations when applied to convergently non-stationary dynamics, such as the ones arising in a feudal hierarchical setting. Additionally, we establish convergence for Q -learning-based deep reinforcement learning methods with convergently non-stationary features, such as the ones arising in representation-based settings. Our findings offer theoretical support for the application of Q -learning in these complex scenarios and present methodologies for extending established theoretical results from standard cases to their convergently non-stationary counterparts.

We study hybrid execution in multi-agent reinforcement learning (MARL), a paradigm where agents aim to complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information.

State-of-the-art agent-modelling tools support the creation of powerful Socially Intelligent Agents (SIAs) capable of engaging in social interactions with participants in various roles and environments. However, their deployment demands a labourious authoring task as it is necessary to manually define behaviour rules and create content for different interaction scenarios. While Socially Intelligent Agents (SIAs) research has centred on the user experience, we shift focus to the authors. To understand the challenges faced by authors who create these agents, we performed an innovative analysis of the authoring experience in modern agent modelling tools. One key finding is that, while SIA concepts are generally understandable, emotional-based concepts are not as easily comprehended or used by authors. We propose a hybrid solution approach that culminated in the development of Authoring-Assisted FAtiMA-Toolkit. The augmented agent modelling tool incorporates a data-driven Authoring Assistant to boost author productivity while promoting transparency and authorial control. To evaluate the impact of this framework on the authoring experience, we conducted a user study. Results showed that authors using the Authoring-Assisted FAtiMA-Toolkit were on average able to create more SIA-related content in less time. Our findings suggest that data-augmented, theory-grounded agent modelling tools can support the development of affective social agents by reducing the authoring burden without sacrificing the framework’s clarity or the authors’ control over the content.

Hierarchical reinforcement learning is an increasingly demanded resource for learning to make sequential decisions towards long term goals. Feudal hierarchies are among the most deployed frameworks. However, there are few theoretical results for hierarchical structures. In this work, we formalize the common two-level feudal hierarchy as two Markov decision processes, with the one on the high level being dependent on the policy executed at the low level. Despite the non-stationarity raised by the dependency, we show that each of the processes presents stable behavior. We then build on the first result to show that, regardless of the convergent learning algorithm used for the low level, convergence of both prediction and control algorithms at the high-level is guaranteed. Our results contribute with theoretical support for the use of feudal hierarchies in combination with standard reinforcement learning methods at each level.

Traditionally, video-game maps are either made by hand, requiring many man-hours to produce good results, or made using Procedural Content Generation (PCG) techniques, which rely on a predetermined algorithm to generate every feature of the map. More recent studies have tried an approach using Deep Learning algorithms, which have their own limitations, in particular taking away the creative freedom of the designers. To circumvent this problem we propose a system that transforms low fidelity sketches into realistic height-maps through a Deep Learning model we call the Dual Critic Conditional Wasserstein GAN (DCCWGAN), thus providing high visual quality without removing control from the user. The presented system is capable of producing images that resemble the received input, and a user study with 79 participants showed that observers are not able to distinguish between earth-based height-map images and the images generated by our system.

Current Higher Education Institutions' mental health support systems lack the capabilities to cope with the growing need and demand for mental health support from students. We introduce MHeVA -- Mental Health Virtual Assistant -- which was designed with the goal of creating an intelligent virtual agent that could serve as a first-line diagnostic-aid tool for mental health services across universities and faculties. Students interact with the agent which attempts to establish rapport and promotes disclosure through mental health state evaluation questions. In addition to this, MHeVA has the ability to assess self-reported anxiety levels, provide health improvement tips and flag the most severe cases.

Socially Intelligent Agents (SIAs) have become increasingly popular in various contexts, including education and entertainment. However, creating complex social scenarios tailored to a designer's specific goals remains a significant challenge. The authoring burden can be substantial, limiting the potential of SIAs to deliver rich, engaging experiences. In this work, we propose leveraging the extensive knowledge stored within Large Language Models and use theory-driven prompting to extract social practices and identify appropriate social affordances for a scenario description. Our prompting approach aims to guide the system into considering the essential components (beliefs and desires) necessary to produce intentions, actions, and emotions1. Results show that our approach produces large amounts of accurate and new information that can add value to the scenario. However, the process can introduce inaccuracies without human supervision.

Style and emotional expressiveness are essential aspects of virtual character computer animation. For a virtual character to display different emotions, motion capture data conveying each desired style has to be recorded, even if the baseline motion is the same. Animators then have to refine and conjoin each recording in order to create the final animations making it a timely and costly process. Although there have been efforts made into the automatic generation of motions, the problem persists that, for each new desired emotion, reference data displaying said emotion has to be readily available and a new motion has to be learned from scratch. By combining Machine Learning with Emotion Analysis - in particular Laban Movement Analysis and the Pleasure, Arousal, Dominance Emotional State Model - we have developed a system that is capable of not only identifying the perceived emotion of locomotion animations but that also allows users to alter the character’s expressed emotion in real time and without the need of additional data.

More than a decade has passed since the development of FearNot!, an application designed to help children deal with bullying through role-playing with virtual characters. It was also the application that led to the creation of FAtiMA, an affective agent architecture for creating autonomous characters that can evoke empathic responses. In this article, we describe the FAtiMA Toolkit, a collection of open-source tools that is designed to help researchers, game developers, and roboticists incorporate a computational model of emotion and decision-making in their work. The toolkit was developed with the goal of making FAtiMA more accessible, easier to incorporate into different projects, and more flexible in its capabilities for human-agent interaction, based upon the experience gathered over the years across different virtual environments and human-robot interaction scenarios. As a result, this work makes several different contributions to the field of Agent-Based Architectures. More precisely, the FAtiMA Toolkit’s library-based design allows developers to easily integrate it with other frameworks, its meta-cognitive model affords different internal reasoners and affective components, and its explicit dialogue structure gives control to the author even within highly complex scenarios. To demonstrate the use of the FAtiMA Toolkit, several different use cases where the toolkit was successfully applied are described and discussed.

Social sharing of emotions (SSE) occurs when one communicates their feelings and reactions to a certain event in the course of a social interaction. The phenomenon is part of our social fabric and plays an important role in creating empathetic responses and establishing rapport. Intelligent social agents capable of SSE will have a mechanism to create and build long-term interaction with humans. In this paper, we present the Emotional Episode Generation (EEG) model, a fine-tuned GPT-2 model capable of generating emotional social talk regarding multiple event tuples in a human-like manner. Human evaluation results show that the model successfully translates one or more event-tuples into emotional episodes, reaching quality levels close to human performance. Furthermore, the model clearly expresses one emotion in each episode as well as humans. To train this model we used a public dataset and built upon it using event extraction techniques1.

In this work we test the hypothesis that interacting with an intelligent virtual character in Virtual Reality (VR) has a stronger impact compared to the same interaction in a traditional non-immersive platform, both in terms of presence and believability.

In this work, we identify a novel set of conditions that ensure convergence with probability 1 of Q-learning with linear function approximation, by proposing a two time-scale variation thereof. In the faster time scale, the algorithm features an update similar to that of DQN, where the impact of bootstrapping is attenuated by using a Q-value estimate akin to that of the target network in DQN. The slower time-scale, in turn, can be seen as a modified target network update. We establish the convergence of our algorithm, provide an error bound and discuss our results in light of existing convergence results on reinforcement learning with function approximation. Finally, we illustrate the convergent behavior of our method in domains where standard Q-learning has previously been shown to diverge.

This paper presents the results of a user study designed to investigate social robots' persuasiveness. In the design, the robot attempts to persuade users in two different conditions comparing to a control condition. In one condition, the robot aims at persuading users by giving them a reward. In the second condition, the robot tries to persuade by punishing users. The results indicated that the robot succeeded to persuade the users to select a less-desirable choice comparing to a better one. However, no difference was found in the perception of the robot's warmth nor discomfort, comparing the two strategies. The results suggest that social robots are capable of persuading users objectively, but further investigation is required to investigate persuasion subjectively.

In this paper, we propose a feature-based model to recognize emotions via touching patterns of individuals playing a game on a typical tablet. In this work, novel features, such as Angular Velocity/Acceleration, Angle, Curl, Area and number of strokes within a time window, are introduced and the gold-standard of the data is determined automatically via subjects' facial expressions. The results show that the approach is promising and the model is able to recognize all the six basic emotions, with a performance of 71.92 % ±0.51. In addition, the recognition of valence and arousal reaches correlation coefficients equal to 0.76 and 0.78 respectively.

Even though modern video games present rich storytelling and high graphical fidelity, they are still lacking in rich non-scripted NPC social interactions. In this work we present an extension of the Comme il Faut (CiF) and CiF-CK social interaction models, where we added emotions and a emotion appraisal process based on the OCC model of emotion and also added a belief system that models the social network values that the NPCs expect regarding the other NPCs' relationships. A version of the new model, which we named Comme il Faut - Exiles (CiF-Ex), was implemented in the AAA game Conan Exiles using their modding tools and validated. The results were noteworthy: the users enjoyed the game more and tended to spend more time near CiF-Ex enabled NPCs. The majority of the users preferred CiF-Ex NPCs, versus the vanilla NPCs. CiF-Ex NPCs were perceived as more believable and less predictable.

In this work we propose and test a new model for socially intelligent merchants to be used in Role Playing Games. The objective is to enhance the player's experience in terms of immersion when interacting with merchants and increase the merchants' believability as Non-Player Characters, while minimizing the authoring effort. The model is based on the CIF architecture, with merchant specific components added. The model allows the creation of different kinds of merchants based on their traits that convey different personalities with an arrangement of different possible interactions, all this without much effort. An abbreviated version of the model was implemented and tested in the game Conan Exiles. The results were very positive, with merchants using our model proving to be significantly more enjoyable and believable when compared with a merchant without the model. In the end, there was an almost unanimous preference by the players for interacting with the merchants that had our model in relation to the one without it.

Recent advances on Social Robotics raise the question whether a social robot can be used as a persuasive agent. To date, a body of literature has been performed using various approaches to answer this research question, ranging from the use of non-verbal behavior to the exploration of different embodiment characteristics. In this paper, we investigate the role of social power for making social robots more persuasive. Social power is defined as one's ability to influence another to do something which s/he would not do without the presence of such power. Different theories classify alternative ways to achieve social power, such as providing a reward, using coercion, or acting as an expert. In this work, we explored two types of persuasive strategies that are based on social power (specifically Reward and Expertise) and created two social robots that would employ such strategies. To examine the effectiveness of these strategies we performed a user study with 51 participants using two social robots in an adversarial setting in which both robots try to persuade the user on a concrete choice. The results show that even though each of the strategies caused the robots to be perceived differently in terms of their competence and warmth, both were similarly persuasive.

Social Power, a pervasive feature in our daily life, has been proved to have a significant impact on Social Interaction; While the capability of maintaining a Social Interaction has an acknowledged role in Believability of Intelligent Virtual Agents (IVAs). In this paper, we argue that an ability of reasoning and planning in the presence of Social Power enhances Social Believability of IVAs, leading to more rational interactions. With this aim, we focus on theoretical issues of agent modeling aiming at increasing intelligence and therefore believability of NPCs or agents of game-like simulations or serious games. Thereby, we propose a model of social power inspired by a recently proposed model, SAPIENT, based on a well-known theory of Social Power proposed by French and Raven.

Social power is a pervasive feature with acknowledged impact in a multitude of social processes. However, despite its importance, common approaches to social power interactions in multi-agent systems are rather simplistic and lack a full comprehensive view of the processes involved. In this work, we integrated a comprehensive model of social power dynamics into a cognitive agent architecture based on an operationalization of different bases of social power inspired by theoretical background research in social psychology. The model was implemented in an agent framework that was subsequently used to generate the behavior of virtual characters in an interactive virtual environment. We performed a user study to assess users' perceptions of the agents and found evidence supporting both the social power capabilities provided by the model and their value for the creation of believable and interesting scenarios. We expect that these advances and the collected evidence can be used to support the development of agent systems with an enriched capacity for social agent simulation.