This paper describes the development of a system to play an ancient Basque collaborative percussion instrument (the txalaparta) with a human performer. A deep probabilistic model was trained on two large datasets, and transfer learning used to train the model on high-quality data abstracted from txalaparta performances. The aim is to create a system that effectively responds to the human player in a performance environment, where the model's response must feel instantaneous, organic, and sensible in response to the user's input. The paper outlines the design and implementation of the system, including data collection, model training, and performance visualization. The study contributes to the field of interactive music systems and demonstrates the potential of deep learning in creating intelligent musical systems that can collaborate with human performers.

When people form impressions of others in face-to-face communication, gesture style (i.e. the way of gesturing) impacts their impressions, such as being well-mannered, honest, and enthusiastic. As a mechanism for changing the gesture style, we trained a GAN-based style transfer model using a collection of video clips of speakers. Then, we collected a new speaker dataset from YouTube videos representing three different countries, and applied them to the style encoder of our style transfer model and created a gesture style latent space. However, it is difficult to select an appropriate style from a large number of style candidates to synthesize motions that would give off a specific impression. To assist users with this, we propose a method for automatically selecting an appropriate style by fine-tuning a Large Language Model (LLM) that uses a list of impression words as input. An evaluation study found that the gesture transfer model effectively changes the impression of gesture, and styles selected by the style selection mechanism produced motions that express similar impression to those that applied ground truth styles, compared to randomly selected styles.

Crowd simulation is striving to advance the realism of large groups of intelligent virtual agents. There have been several efforts to create common frameworks which can expedite collaboration among researchers. In this paper, we propose three extensions to one such framework called Menge, making it easier to use with Unity 3D.

Sex crimes and aggravated assaults are some of the most difficult cases to be tried in court, especially for the victims that need to take the witness stand in the presence of the accused, the judge and other participants of the proceedings. Victims are already strained by the nature of their situation, and the additional burden of needing to face their alleged perpetrator again under unfamiliar circumstances and become the focus of everyone's attention, can become overwhelming and cause severe anxiety. Today, courtrooms are sketched on paper to prepare the victims, e.g. by showing them a seating arrangement, but the effectiveness of this method has been questioned. Therefore, an ongoing pilot project, in close collaboration with the police, the court system and psychologists working with young victims of sexual assault, is exploring the use of an interactive court proceedings simulation in VR. The simulation involves a recreation of the exact location of the planned proceedings, as well as some of the social factors, using virtual humans. This extended abstract describes work in progress, focusing on the motivation for doing this, why this might be an effective approach, and a working prototype.

The ability to recognize social cues and knowing how to approach people in public is an important skill. SoCueVR is a virtual reality game, intended to teach children with autism how to detect social cues and knowing how to initiate contact with strangers. The goal of the game is to support a charity by selling pens to people walking around a mall. Only some of the virtual people are interested in buying and that can be seen in their nonverbal behavior, such as gaze and facial expression. By detecting the nonverbal cues early, the player spends more time selling and less time getting turned down. A prototype was user tested and reviewed by an expert in social training for children with autism, suggesting strong potential.

Many virtual environments rely on a social population that needs to behave in a plausible manner. Crowd simulations typically concern themselves with simulating collision free movement, while getting into and managing social contact with other humans is a less explored subject. Complex situations such as forming conversation groups and recognizing each other presence need to be tackled. This study presents a typical real-life scenario at a university, which in spite of its mundane nature uncovers a wide range of simulation challenges. We apply our reactive rule-based system that models territoriality, proxemics and social navigation to this scenario to demonstrate a possible approach to addressing some of these challenges and contribute to a vocabulary for social simulation.

Awareness of the surroundings as well as bodily cues is essential for effective social interaction. Nonverbal behavior such as gaze, facial expression, gesture and posture, provide cues for interpreting intent and relational status. These cues are also present as people traverse a physical setting, where locomotion and social expression integrate effortlessly. The goal of this study was to examine how human observers interpreted social behavior of virtual agents constrained by the physical environment of a movie theater with seating rows. An online survey compared videos of agents exhibiting only locomotion, and agents that also exhibited two different levels of social commitment. Results showed that people were able to interpret the social behavior correctly, indicating that adding it on top of complex maneuvering is possible. These results continue to build a bridge between the literature on agent navigation and agent social behavior.

In the everyday world people form small conversing groups where social interaction takes place, and much of the social behavior takes place through managing interpersonal space (i.e., proxemics) and group formation, signaling their attentio to others (i.e., through gaze behavior), and expressing certain attitudes, for example, friendliness, by smiling, getting close through increased engagement and intimacy, and welcoming newcomers. Many real-time interactive systems feature virtual anthropomorphic characters in order to simulate conversing groups and add plausibility and believability to the simulated environments. However, only a few have dealt with autonomous behavior generation, and in those cases, the agents’ exhibited behavior should be evaluated by users in terms of appropriateness, believability, and conveyed meaning (e.g., attitudes). In this article we present an integrated intelligent interactive system for generating believable nonverbal behavior exhibited by virtual agents in small simulated group conversations. The produced behavior supports group formation management and the expression of interpersonal attitudes (friendly vs. unfriendly) both among the agents in the group (i.e., in-group attitude) and towards an approaching user in an avatar-based interaction (out-group attitude). A user study investigating the effects of these attitudes on users’ social presence evaluation and proxemics behavior (with their avatar) in a three-dimensional virtual city environment is presented. We divided the study into two trials according to the task assigned to users, that is, joining a conversing group and reaching a target destination behind the group. Results showed that the out-group attitude had a major impact on social presence evaluations in both trials, whereby friendly groups were perceived as more socially rich. The user’s proxemics behavior depended on both out-group and in-group attitudes expressed by the agents. Implications of these results for the design and implementation of similar intelligent interactive systems for the autonomous generation of agents’ multimodal behavior are briefly discussed.

In greeting encounters, first impressions of personality and attitude are quickly formed and might determine important relational decisions, such as the likelihood and frequency of subsequent encounters. An anthropomorphic user interface is not immune to these judgments, specifically when exhibiting social interaction skills in public spaces. A favorable impression may help engaging users in interaction and attaining acceptance for long-term interactions. We present three studies implementing a model of first impressions for initiating user interactions with an anthropomorphic museum guide agent with socio-relational skills. We focus on nonverbal behavior exhibiting personality and interpersonal attitude. In two laboratory studies, we demonstrate that impressions of an agent's personality are quickly formed based on proximity, whereas interpersonal attitude is conveyed through smile and gaze. We also found that interpersonal attitude has greater impact than personality on the user's decision to spend time with the agent. These findings are then applied to a museum guide agent exhibited at the Boston Museum of Science. In this field study, we show that employing our model increases the number of visitors engaging in interaction.

Path following in games mostly focuses on avoiding collisions with dynamic physical objects that appear along a chosen path to a given destination. Some work also attempts to humanize the abstract path returned by a path finding algorithm through methods like smoothing. Games typically do not consider social factors during path following, even though many depict social environments. Social path following considers the social environment in particular, carving a trajectory that reflects awareness of other human beings and their social activities. This includes awareness of territories that have social significance but no concrete physical form, such as the space between those having a conversation. This paper describes work that extends a state-of-the-art predictive method for path following with social awareness, predicting and avoiding social collisions. The work builds on a platform for social simulation, which already models social territoriality and gaze behavior. The results appear promising and highlight the importance of perceiving and dealing with the social space along with the physical one.

Abstract Work on levels of detail for human simulation has occurred mainly on a geometrical level, either by reducing the numbers of polygons representing a virtual human, or replacing them with a two‐dimensional imposter. Approaches that reduce the complexity of motions generated have also been proposed. In this paper, we describe ongoing development of a framework for Adaptive Level Of Detail for Human Animation (ALOHA), which incorporates levels of detail for not only geometry and motion, but also includes a complexity gradient for natural behaviour, both conversational and social. ACM CSS: I.3.7 Three‐Dimensional Graphics and Realism—Animation

The Behavior Expression Animation Toolkit (BEAT) allows animators to input typed text that they wish to be spoken by an animated human figure, and to obtain as output appropriate and synchronized nonverbal behaviors and synthesized speech in a form that can be sent to a number of different animation systems. The nonverbal behaviors are assigned on the basis of actual linguistic and contextual analysis of the typed text, relying on rules derived from extensive research into human conversational behavior. The toolkit is extensible, so that new rules can be quickly added. It is designed to plug into larger systems that may also assign personality profiles, motion characteristics, scene constraints, or the animation styles of particular animators.

Prior research into embodied interface agents has found that users like them and find them engaging. In this paper, we argue that embodiment can serve an even stronger function if system designers use actual human conversational protocols in the design of the interface. Communicative behaviors such as salutations and farewells, conversational turn-taking with interruptions, and referring to objects using pointing gestures are examples of protocols that all native speakers of a language already know how to perform and that can thus be leveraged in an intelligent interface. We discuss how these protocols are integrated into Rea, an embodied, multi-modal conversational interface agent who acts as a real-estate salesperson, and we show why embodiment is required for their successful implementation.

In this paper, we argue for embodied corrversational characters as the logical extension of the metaphor of human - computer interaction as a conversation. We argue that the only way to fully model the richness of human I&+ to-face communication is to rely on conversational analysis that describes sets of conversational behaviors as fi~lfilling conversational functions, both interactional and propositional. We demonstrate how to implement this approach in Rea, an embodied conversational agent that is capable of both multimodal input understanding and output generation in a limited application domain. Rea supports both social and task-oriented dialogue. We discuss issues that need to be addressed in creating embodied conversational agents, and describe the architecture of the Rea interface.