Public speaking experts intentionally take their audience on an emotional roller coaster, staying attuned to their audience's collective emotional feedback. In this research, we explore how bidirectional sharing of heart rates between a speaker and their audience facilitates this emotional exchange, through empathy, emotional awareness, and engagement. Firstly, in two design studies$(N=25)$, we evaluated this concept and identified design elements for a heart rate sharing interface. Subsequently, we developed Feeling-the-Beat, a system for sharing heart rate between speakers and audiences in real time. Finally, in a randomized, counter-balanced, within-subjects study$(N=36)$, we compared our system to one sharing fabricated heart rates and a baseline system without heart rate sharing. Feeling-the-Beat significantly increased audience empathy towards the speaker, audience engagement during moments of heightened speaker heart rate, and social presence.

We introduce a new approach to deploying constraint-based content generators that better supports online generation. Constraint-based generators ensure that certain properties hold in each design they output. However, when deployed a general-purpose solver is often required, thus guarantees come with unpredictable search times and little control over sequentially-generated outputs. In this paper, we outline how we can encode design constraints into a compact circuit representation that affords generation without search. These generators yield samples that are distributed uniformly over the space of valid designs. We illustrate our approach with binary decision diagrams (BDDs) in comparison to the traditional approach with answer-set programming (ASP) in two scenarios: a grid-based tile placement scenario inspired by WaveFunctionCollapse, and a playable platformer level design scenario. These compiled design-space models make constraint-based methods easier to deploy by improving on both the running time and diversity of previous constraint-based methods.

With the advancement of Large Language Models (LLMs), they are increasingly being used as a backend for interactive virtual agents and assistants. Therefore, a critical social skill for these agents is Theory of Mind (ToM): the ability to model and reason about other agents. Research has investigated ToM in LLMs using standard, modified, and extended versions of false-belief tasks. These tests include explicit prompts asking LLMs to answer questions about other agents. However, in real situations, people have to use ToM unprompted to navigate social life. Additionally, oftentimes, people have to rely on nonverbal cues such as facial expressions. This work seeks to address this gap by studying implicit ToM in LLMs in a negotiation task. In negotiation, agents have to implicitly reason about other agents to reach an agreed-upon best possible deal. We conducted the negotiation experiment by prompting different LLMs to roleplay as characters and pitting them against rule-based agents that may respond with different facial expressions. We measure and compare the outcomes of the negotiation across models. Our results show that strong LLMs like GPT-4 turbo and Claude 3 Opus can perform decently and adjust their offers based on access to facial expression information, but weaker models are far behind. Our work contributes to our understanding of LLMs’ capabilities and limitations for serving as intelligent and interactive agents.

Studying mental models has recently received more attention, aiming to understand the cognitive aspects of human-computer interaction. However, there is not enough research on the elicitation of mental models in complex dynamic systems. We present Thought Bubbles as an approach for eliciting mental models and an avenue for understanding players’ mental model development in interactive virtual environments. We demonstrate the use of Thought Bubbles in two experimental studies involving 250 participants playing a supply chain game. In our analyses, we rely on Situation Awareness (SA) levels, including perception, comprehension, and projection, and show how experimental manipulations such as disruptions and information sharing shape players’ mental models and drive their decisions depending on their behavioral profile. Our results provide evidence for the use of thought bubbles in uncovering cognitive aspects of behavior by indicating how disruption location and availability of information affect people’s mental model development and influence their decisions.

Abstract We stumble upon new and repeating information daily. As information comes from many sources, social media continues to play a predominant role in disseminating information, ultimately impacting individuals’ perceptions and behaviors. A prime example of this impact was observed during the COVID-19 pandemic, in which social media use was influencing willingness to receive the COVID-19 vaccine. While studies on this relationship between social media use and vaccination intent have been widely investigated, less is known about the mechanisms that link these two variables, specifically the types of information seen on social media platforms and the effects of these different types of information. In this exploratory study, we demonstrate the mediator role of information exposure (to include both types of information and frequency) between social media use and vaccination intent. Our results show that different types of information mediate this relationship differently and demonstrate how these relationships were further moderated by the income level of the participant. We conclude with the implications of these findings and how our findings can inform the direction of future research within the field of human–computer interaction.

Virtual humans and social robots frequently generate behaviors that human observers naturally see as expressing emotion. In this review article, we highlight that these expressions can have important benefits for human–machine interaction. We first summarize the psychological findings on how emotional expressions achieve important social functions in human relationships and highlight that artificial emotional expressions can serve analogous functions in human–machine interaction. We then review computational methods for determining what expressions make sense to generate within the context of interaction and how to realize those expressions across multiple modalities, such as facial expressions, voice, language, and touch. The use of synthetic expressions raises a number of ethical concerns, and we conclude with a discussion of principles to achieve the benefits of machine emotion in ethical ways.

People often encounter highly stressful, emotion-evoking situations. Modeling and predicting people's behavior in such situations, how they cope, is a critical research topic. To that end, we propose a computational model of coping that casts Lazarus's theory of coping into a Partially Observable Markov Decision Process (POMDP) framework. This includes an appraisal process that models the factors leading to stress by assessing a person's relation to the environment and a coping process that models how people seek to reduce stress by directly altering the environment or changing one's beliefs and goals. We evaluated the model's assumptions in the context of a high-stress situation, hurricanes. We collected questionnaire data from major U.S. hurricanes in 2018 to evaluate the model's features for appraisal calculation. We also conducted a series of controlled experiments simulating a hurricane experience to investigate how people change their beliefs and goals to cope with the situation. The results support the model's assumptions showing that the proposed features are significantly associated with the evacuation decisions and people change their beliefs and goals to cope with the situation.

People experience many stressful, emotion-evoking situations in everyday life. How they cope with these situations is crucial to their well-being. Research shows that people may change their beliefs to perceive the situations in a better, less-stressful light. Therefore, understanding how people change their beliefs to cope with stress and emotion is an important research question. Toward that end, we model coping, based on Lazarus's appraisal theory of emotion, as a two-step decision problem. However, people do not hallucinate arbitrary alternative realities in order to reduce stress. So a central challenge here is to model the constraints on belief change. We specifically stipulate two key factors influencing the degree of belief change: the utility of holding the alternative belief and constraints on changing a belief based on its underlying uncertainty. To investigate these factors and the model's assumptions, we applied the model to a simple hurricane situation and conducted an experiment based on this situation, where participants observed hurricane information and reported their beliefs about it. We found that when the hurricane worsens, those who stayed believe the hurricane to be less severe than the most likely outcome from the information and those who evacuated. The results also show that the uncertainty of information and the utility of the beliefs about that information is related and together determine how emotion-focused coping alters the person's beliefs. Overall, the results support the assumptions and predictions of the model. These findings illustrate the relevance of applying a decision-making model analysis to coping.

We look at the possibilities of automatically detecting social discomfort and social anxiety via non-verbal behaviours in social Virtual Reality (VR). This is important because a well-developed automatic recognition system could facilitate interventions and moderation in social VR without requiring real-time parental supervision. To initially explore this question of recognition, we prototyped a small set of 3D stimuli representing a bullying scenario and explored in a small formative preliminary study what human observers perceived from the stimuli. Future work is required with different problematic situations in social VR and evaluations with more participants before developing an automatic recognition system.

Understanding decision-making in dynamic and complex settings is a challenge yet essential for preventing, mitigating, and responding to adverse events (e.g., disasters, financial crises). Simulation games have shown promise to advance our understanding of decision-making in such settings. However, an open question remains on how we extract useful information from these games. We contribute an approach to model human-simulation interaction by leveraging existing methods to characterize: (1) system states of dynamic simulation environments (with Principal Component Analysis), (2) behavioral responses from human interaction with simulation (with Hidden Markov Models), and (3) behavioral responses across system states (with Sequence Analysis). We demonstrate this approach with our game simulating drug shortages in a supply chain context. Results from our experimental study with 135 participants show different player types (hoarders, reactors, followers), how behavior changes in different system states, and how sharing information impacts behavior. We discuss how our findings challenge existing literature.

During crises like COVID-19, individuals are inundated with conflicting and time-sensitive information that drives a need for rapid assessment of the trustworthiness and reliability of information sources and platforms. This parallels evolutions in information infrastructures, ranging from social media to government data platforms. Distinct from current literature, which presumes a static relationship between the presence or absence of trust and people’s behaviors, our mixed-methods research focuses on situated trust, or trust that is shaped by people’s information-seeking and assessment practices through emerging information platforms (e.g., social media, crowdsourced systems, COVID data platforms). Our findings characterize the shifts in trustee (what/who people trust) from information on social media to the social media platform(s), how distrust manifests skepticism in issues of data discrepancy, the insufficient presentation of uncertainty, and how this trust and distrust shift over time. We highlight the deep challenges in existing information infrastructures that influence trust and distrust formation.

Increasingly, PCG systems are developed to help game designers create content for their games. However, game designers have limited control over the content current PCG systems generate. We investigate an interaction mechanism non-technical users can use to control generative grammars without the need for understanding the grammar’s rules. To demonstrate this control mechanism, we present a system, built using a probabilistic graph grammar, that allows designers to specify their desired generative space by defining a region on an expressive range plot. We ran a user study with game design students to assess its viability. Our findings suggest that designers have an easier time controlling the grammar using this mechanism over manually interacting with grammars rules.

To assist game developers in crafting game NPCs, we present EvolvingBehavior, a novel tool for genetic programming to evolve behavior trees in Unreal®Engine 4. In an initial evaluation, we compare evolved behavior to hand-crafted trees designed by our researchers, and to randomly-grown trees, in a 3D survival game. We find that EvolvingBehavior is capable of producing behavior approaching the designer’s goals in this context. Finally, we discuss implications and future avenues of exploration for co-creative game AI design tools, as well as challenges and difficulties in behavior tree evolution.

Gestures convey critical information within social interactions. As such, the success of virtual agents (VA) in both building social relationships and achieving their goals is heavily dependent on the information conveyed within their gestures. Because of the precision required for effective gesture behavior, it is prudent to retain some designer control over these conversational gestures. However, in order to exercise that control practically we must first understand how gestural motion conveys meaning. One consideration in this relationship between motion and meaning is the notion of Ideational Units, meaning that only parts of a gesture’s motion at a point in time may convey meaning, while other parts may be held from the previous gesture. In this paper, we develop, demonstrate, and release a set of tools that help quantify the relationship between the semantics conveyed in a gesture’s co-speech utterance and the fine-grained motion of that gesture. This allows us to explore insights into the complex relationship between motion and meaning. In particular, we use spectral motion clustering to discern patterns of motion that tend to be associated with semantic concepts, on both an aggregate and individual-speaker level. We then discuss the potential for these tools to serve as a framework for both automated gesture generation and interpretation in virtual agents. These tools can ideally be used within approaches to automating VA gesture performances as well as serve as an analysis framework for fundamental gesture research.

Game AI designers must manage complex interactions between the AI character, the game world, and the player, while achieving their design visions. Computational co-creativity tools can aid them, but first, AI and HCI researchers must gather requirements and determine design heuristics to build effective co-creative tools. In this work, we present a participatory design study that categorizes and analyzes game AI designers’ workflows, goals, and expectations for such tools. We evince deep connections between game AI design and the design of co-creative tools, and present implications for future co-creativity tool research and development.

Gestures play a vital role in face-to-face interactions, from conveying speaker attitudes to relaying information not present in speech. Gestures have been widely shown to be linked to the meaning, form and timing of co-speech context of their production. Producing convincing, relevant, and informative semantic gestures is an ongoing challenge in the field of gesture generation for embodied conversational agents. In this paper, we put forward a novel technique to select semantically-related gestures from a gesture database, and present two experiments which highlight the importance of measuring the qualitative impact of semantically-related gestures on the viewer. In the first experiment, we demonstrate a strong correlation between subjective perception of energy level of the speaker and perception of the semantic relatedness of the co-speech transcript. In the second experiment, we attempt to measure semantic information conveyed in gesture on specific qualitative dimensions. We then discuss the implications and impacts of these findings, including the limitations of the strength of claims we can make when using the original gesture that accompanies an utterance as an evaluative baseline.

Agent-based simulations are widely used for modeling human behavior in various contexts. However, such simulations may oversimplify human decision-making. We propose the use of Gamettes to extract rich data on human decision-making and help in improving the human behavioral aspects of models underlying agent-based simulations. We show how Gamettes are designed and provide empirical validation for using Gamettes in an experimental supply chain setting to study human decision-making. Our results show that Gamettes are successful in capturing the expected behaviors and patterns in supply chain decisions, and, thus, we find evidence for the capability of Gamettes to inform behavioral models.

One potential solution to help people change their eating behavior is to develop conversational systems able to recommend healthy recipes. Beyond the intrinsic quality of the recommendations themselves, various factors might also influence users? perception of a recommendation. Two of these factors are the conversational skills of the system and users' interaction modality. In this paper, we present Cora, a conversational system that recommends recipes aligned with its users? eating habits and current preferences. Users can interact with Cora in two different ways. They can select predefined answers by clicking on buttons to talk to Cora or write text in natural language. On the other hand, Cora can engage users through a social dialogue, or go straight to the point. We conduct an experiment to evaluate the impact of Cora's conversational skills and users' interaction mode on users' perception and intention to cook the recommended recipes. Our results show that a conversational recommendation system that engages its users through a rapport-building dialogue improves users' perception of the interaction as well as their perception of the system.

To build agents that can engage users in more open-ended social contexts, research has increasingly been focused on data-driven approaches to reduce the requirement of extensive, hand-authored behavioral content creation. However, one fundamental challenge of data-driven approaches is acquiring the interaction data with sufficient variety that reflects the characteristics of open-ended social interactions. Previous work attempts to acquire social interaction data either from face-to-face interactions or human-agent interactions using a simulated environment. In this work, Active Analysis (AA), a theater rehearsal technique, was applied to collect diverse social strategies and interactions. In particular, this work integrated AA into a web-based crowdsourcing task that requires two crowd workers to conduct a bilateral multi-level multi-issue negotiation. Findings from a between-subject experiment with 200 crowd workers recruited from Amazon Mechanical Turk demonstrated that AA could facilitate the creativity of crowd workers and thus lead to social interaction data with greater variety. In addition, AA provides a means to control the diversity so that the coverage of the collected data is consistent with the goals of the application. The results presented in the paper lay a good foundation for future work on data-driven approaches to build socially interactive agents.

One of the main challenges for conversational agents is to select the optimal dialogue policy based on the state of the interaction. This challenge becomes even harder when the conversational agent not only has to achieve a specific task, but also aims at building rapport. Although some work already tried to tackle this challenge using a Reinforcement Learning (RL) approach, they tend to consider one single optimal policy for all the users, regardless of their conversational goals. In this work, we describe a framework that allows us to build a RL-based agent able to adapt its dialogue policy depending on its user's conversational goals. After we build a rule-based agent and a user simulator communicating at the dialog-act level, we crowdsource the surface sentences authoring for both the simulated users and the agent, which allow us to generate a dataset of interactions in natural language. Then, we annotate each of these interactions with a single rapport score and analyze the links between simulated users' conversational goals, agent conversational policies, and rapport. Our results show that rapport was higher when both or none of the interlocutors tried to build rapport. We use this result to inform the design of a social reward function, and we rely on this social reward function to train a RL-based agent using an hybrid approach of supervised learning and reinforcement learning. We evaluate our approach by comparing two different versions of our RL-based agent: one that takes users' conversational goals into account and another that does not. The results show that an agent adapting its dialogue policy depending on users' conversational goals performs better.

The use of metaphoric gestures by speakers has long been known to influence thought in the viewer. What is less clear is the extent to which the expression of multiple metaphors in a single gesture reliably affect viewer interpretation. Additionally, gestures which express only one metaphor are not sufficient to explain the broad array of metaphoric gestures and metaphoric scenes that human speakers naturally produce. In this paper we address three issues related to the implementation of metaphoric gestures in virtual humans. First, we break down naturally occurring examples of multiple-metaphor gestures, as well as metaphoric scenes created by gesture sequences. Then, we show the importance of capturing multiple metaphoric aspects of gesture with a behavioral experiment using crowdsourced judgements of videos of alterations of the naturally occurring gestures. Finally, we discuss the challenges for computationally modeling metaphoric gestures that are raised by our findings.

The development of agents that can engage in human social interaction has become critical in an increasingly wide range of applications. The focus of this work is modeling agents for social skills training where learners can interact with the autonomous characters in social scenarios and thereby acquire skills that can be applied in the real world. A key goal in the design of these systems is to allow users to explore various actions and tactics while still having the agents generate consistent and diverse responses. Providing the ability to explore different tactics raises a significant content challenge for the design of agents. To tackle the creative content creation problem, this paper introduces a humans-in-the-loop iterative process to automatically generate rich, varied content from a small amount of vignettes provided by online crowd workers. Specifically, this process uses the crowd to iteratively refine and improve an ensemble of generative models. The results show that the iterative, ensemble based approach generates more coherent and novel interactions than alternative non-ensemble, non-iterative approaches. The results presented in this paper can potentially provide the basis for flexible agent-based training systems.

Traditional personality assessment techniques often rely on subjective report obtained from questionnaires. This work complements traditional techniques by exploring objective measures of traits at the behavior level. We explored behavior features extracted from smartphone sensing data, and used selected features to predict the traits of the Five Factor Model. The specific dataset we explored was the StudentLife dataset. We found behavior features corresponding to each trait, and were able to predict the traits with varying degrees of accuracy. The best result of each trait are: Extraversion (91.2%), Agreeableness (67.6%), Conscientiousness (70.6%), Neuroticism (79.4%), Openness(73.5%). Our results suggest that behavioral measures extracted from smartphone sensing data has potential in the assessment of personality.

Negotiation is a fundamental aspect of social interaction. Our research aims to contribute towards the creation of artificial agent negotiators that can be used for training purposes to improve human negotiation skills. To achieve that, we address the challenge of identifying differences in human negotiation styles and relating those differences to individuals' personality traits. In particular, we follow a data-driven approach by collecting data on how people negotiate against an agent using a fixed-response strategy during a task involving the partition of a set of items. We then use different machine learning techniques to: 1) analyze the relationship between negotiation styles and personality traits; 2) characterize changes in the human negotiation behavior during the game; 3) discover human behavior patterns in response to different offers by the agent player. Our analyses show how different personality traits lead to distinct behaviors during the negotiation. In turn, this data will allow us to build agent negotiators that have a rich behavioral repertoire and are able to adapt to human negotiation trainees, thus fostering more interesting learning experiences.

Guilt and envy play an important role in social interaction. Guilt occurs when individuals cause harm to others or break social norms. Envy occurs when individuals compare themselves unfavorably to others and desire to benefit from the others’ advantage. In both cases, these emotions motivate people to act and change the status quo: following guilt, people try to make amends for the perceived transgression, and following envy, people try to harm envied others. In this article, we present two experiments that study participants’ experience of guilt and envy when engaging in social decision making with machines and humans. The results showed that, though experiencing the same level of envy, people felt considerably less guilt with machines than with humans. These effects occurred both with subjective and behavioral measures of guilt and envy, and in three different economic games: public goods, ultimatum, and dictator game. This poses an important challenge for human-computer interaction because, as shown here, it leads people to systematically exploit machines, when compared to humans. We discuss theoretical and practical implications for the design of human-machine interaction systems that hope to achieve the kind of efficiency -- cooperation, fairness, reciprocity, etc. -- we see in human-human interaction.

Our method automatically generates realistic nonverbal performances for virtual characters to accompany spo- ken utterances. It analyses the acoustic, syntactic, se- mantic and rhetorical properties of the utterance text and audio signal to generate nonverbal behavior such as such as head movements, eye saccades, and novel gesture animations based on co-articulation.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

The relative influence of perception and situation in emotional judgments has been extensively debated in psychology. A main issue in this debate concerns how these sources of information are integrated. This work proposes a method able to make probabilistic predictions of appraisals of other agents, using mental models of those agents. From these appraisal predictions, predictions about another agent's expressions are made, integrated with observations of the other agent's ambiguous emotional expressions using Bayesian techniques, resulting in updates to the agent's mental models. Our method is inspired by psychological work on human interpretation of emotional expressions. We demonstrate how these appraisals of others' emotions and observations of their expressions can be an integral part of an agent capable of Theory of Mind reasoning.

Appraisal theory is the most influential theory within affective computing, and serves as the basis for several computational models of emotion. The theory makes strong claims of domain-independence: seemingly different situations, both within and across domains are claimed to produce the identical emotional responses if and only if they are appraised the same way. This article tests this claim, and the predictions of a computational model that embodies it, in two very different interactive games. The results extend prior empirical evidence for appraisal theory to situations where emotions unfold and change over time.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. With this demo, we focus our attention on the perception part of the system, a multimodal framework which captures and analyzes user state behavior for both behavioral understanding and interactional purposes. We will demonstrate real-time user state sensing as a part of the SimSensei architecture and discuss how this technology enabled automatic analysis of behaviors related to psychological distress.

The prevalence of negotiation in social interaction has motivated researchers to develop virtual agents that can understand, facilitate, teach and even carry out negotiations. While much of this research has analyzed how to maximize the objective outcome, there is a growing body of work demonstrating that subjective perceptions of the outcome also play a critical role in human negotiation behavior. People derive subjective value from not only the outcome, but also from the process by which they achieve that outcome, from their relationship with their negotiation partner, etc. The affective responses evoked by these subjective valuations can be very different from what would be evoked by the objective outcome alone. We investigate such subjective valuations within human-agent negotiation in four variations of a wartime negotiation scenario taken from the political science literature. We observe that the objective outcomes of these negotiations are not strongly related to the human negotiators' subjective perceptions, as measured by the Subjective Value Index. We examine the game dynamics and agent behaviors to identify features that induce different subjective values in the participants, even when separated from the effect of the objective outcomes. We thus are able to identify characteristics of the interaction between negotiation and battlefield processes that most impact people's subjective valuations.

Recent advances in scanning technology have enabled the widespread capture of 3D character models based on human subjects. However, in order to generate a recognizable 3D avatar, the movement and behavior of the human subject should be captured and replicated as well. We present a method of generating a 3D model from a scan, as well as a method to incorporate a subjects style of gesturing into a 3D character. We present a study which shows that 3D characters that used the gestural style as their original human subjects were more recognizable as the original subject than those that don't.

One significant challenge in creating accurate models of human decision behavior is accounting for the effects of context. Research shows that seemingly minor changes in the presentation of a decision can lead to shifts in behavior, phenomena collectively referred to as framing effects. This work presents a computational modeling analysis comparing the effectiveness of Context Dependent Utility, an appraisal-based approach to modeling the multi-dimensional effects of context on decision behavior, against Cumulative Prospect Theory, Security-Potential/Aspiration Theory, the Transfer of Attention Exchange model, and a power-based utility function. To contrast model performance, a non-linear least-squares analysis and subsequent calculation of Akaike Information Criterion scores, which take into account goodness of fit while penalizing for model complexity, are employed. Results suggest that multi-dimensional models of context and framing, such as Context Dependent Utility, can be much more accurate in modeling decisions which similarly involve multi-dimensional considerations of context. Furthermore, this work demonstrates the effectiveness of employing affective constructs, such as appraisal, for the encoding and evaluation of context within decision-theoretic frameworks to better model and predict human decision behavior.

The prevalence of negotiation in social interaction has motivated researchers to develop virtual agents that can understand, facilitate, teach and even carry out negotiations. While much of this research has analyzed how to maximize the objective outcome, there is a growing body of work demonstrating that subjective perceptions of the outcome also play a critical role in human negotiation behavior. People derive subjective value from not only the outcome, but also from the process by which they achieve that outcome, from their relationship with their negotiation partner, etc. The affective responses evoked by these subjective valuations can be very different from what would be evoked by the objective outcome alone. We investigate such subjective valuations within human-agent negotiation in four variations of a wartime negotiation game. We observe that the objective outcomes of these negotiations are not strongly correlated with the human negotiators' subjective perceptions, as measured by the Subjective Value Index. We examine the game dynamics and agent behaviors to identify features that induce different subjective values in the participants. We thus are able to identify characteristics of the negotiation process and the agents' behavior that most impact people's subjective valuations in our wartime negotiation games.

We demonstrate a lip animation (lip sync) algorithm for real-time applications that can be used to generate synchronized facial movements with audio generated from natural speech or a text-to-speech engine. Our method requires an animator to construct animations using a canonical set of visemes for all pairwise combinations of a reduced phoneme set (phone bigrams). These animations are then stitched together to construct the final animation, adding velocity and lip-pose constraints. This method can be applied to any character that uses the same, small set of visemes. Our method can operate efficiently in multiple languages by reusing phone bigram animations that are shared among languages, and specific word sounds can be identified and changed on a per-character basis. Our method uses no machine learning, which offers two advantages over techniques that do: 1) data can be generated for non-human characters whose faces can not be easily retargeted from a human speaker's face, and 2) the specific facial poses or shapes used for animation can be specified during the setup and rigging stage, and before the lip animation stage, thus making it suitable for game pipelines or circumstances where the speech targets poses are predetermined, such as after acquisition from an online 3D marketplace.

In this special section we report on the workshop Standards in Emotion Modeling held in August 2011, Leiden, The Netherlands. An important goal of this workshop was to identify challenges related to the development and evaluation of computational models of affective processes. These challenges were approached from a psychological and a computational modeling perspective. In this introduction we present a summary of the results of this week-long workshop. In addition to that, we are proud to present an invited contribution that proposes solutions to several of these challenges.

Creating accurate computational models of human decision making is a vital step towards the realization of socially intelligent systems capable of both predicting and simulating human behavior. In modeling human decision making, a key factor is the psychological phenomenon known as "framing", in which the preferences of a decision maker change in response to contextual changes in decision problems. Existing approaches treat framing as a one-dimensional contextual influence based on the perception of outcomes as either gains or losses. However, empirical studies have shown that framing effects are much more multifaceted than one-dimensional views of framing suggest. To address this limitation, we propose an integrative approach to modeling framing which combines the psychological principles of cognitive appraisal theories and decision-theoretic notions of utility and probability. We show that this approach allows for both the identification and computation of the salient contextual factors in a decision as well as modeling how they ultimately affect the decision process. Furthermore, we show that our multi-dimensional, appraisal-based approach can account for framing effects identified in the empirical literature which cannot be addressed by one-dimensional theories, thereby promising more accurate models of human behavior.

During face-to-face conversation, our body is continually in motion, displaying various head, gesture, and posture movements. Based on findings describing the communicative functions served by these nonverbal behaviors, many virtual agent systems have modeled them to make the virtual agent look more effective and believable. One channel of nonverbal behaviors that has received less attention is head movements, despite the important functions served by them. The goal for this work is to build a domain-independent model of speaker's head movements that could be used to generate head movements for virtual agents. In this paper, we present a machine learning approach for learning models of head movements by focusing on when speaker head nods should occur, and conduct evaluation studies that compare the nods generated by this work to our previous approach of using handcrafted rules . To learn patterns of speaker head nods, we use a gesture corpus and rely on the linguistic and affective features of the utterance. We describe the feature selection process and training process for learning hidden Markov models and compare the results of the learned models under varying conditions. The results show that we can predict speaker head nods with high precision (.84) and recall (.89) rates, even without a deep representation of the surface text and that using affective information can help improve the prediction of the head nods (precision: .89, recall: .90). The evaluation study shows that the nods generated by the machine learning approach are perceived to be more natural in terms of nod timing than the nods generated by the rule-based approach.

There has been an increasing interest in exploring how recognition of a user's affective state can be exploited in creating more effective human-computer interaction. It has been argued that IUIs may be able to improve interaction by including affective elements in their communication with the user (e.g. by showing empathy via adequate phrasing of feedback.) This workshop will address a variety of issues related to the development of what we will call the affective loop: detection/modeling of relevant user's states, selection of appropriate system responses (including responses that are designed to influence the user affective state but are not overtly affective), as well as synthesis of the appropriate affective expressions.

No abstract available.

The main factors that affect software understanding are the complexity of the problem solved by the program, the program text, the user's mental ability and experience and the task being performed. The paper describes a planning approach solution to the software understanding problem that focuses on the user's task and expertise. First, user questions about software artifacts have been studied and the most commonly asked questions are identified. These questions are organized into a question model and procedures for answering them are developed. Then, the patterns in user questions while performing certain tasks have been studied and these patterns are used to build generic task models. The explanation system uses these task models in several ways. The task model, along with a user model, is used to generate explanations tailored to the user's task and expertise. In addition, the task model allows the system to provide explicit task support in its interface.