Large language models (LLMs) are increasingly used to simulate human behavior in social settings such as legal mediation, negotiation, and dispute resolution. However, it remains unclear whether these simulations reproduce the personality–behavior patterns observed in humans. Human personality, for instance, shapes how individuals navigate social interactions, including strategic choices and behaviors in emotionally charged interactions. This raises the question: Can LLMs, when prompted with personality traits, reproduce personality-driven differences in human conflict behavior? To explore this, we introduce an evaluation framework that enables direct comparison of human-human and LLM-LLM behaviors in dispute resolution dialogues with respect to Big Five Inventory (BFI) personality traits. This framework provides a set of interpretable metrics related to strategic behavior and conflict outcomes. We additionally contribute a novel dataset creation methodology for LLM dispute resolution dialogues with matched scenarios and personality traits with respect to human conversations. Finally, we demonstrate the use of our evaluation framework with three contemporary closed-source LLMs and show significant divergences in how personality manifests in conflict across different LLMs compared to human data, challenging the assumption that personality-prompted agents can serve as reliable behavioral proxies in socially impactful applications. Our work highlights the need for psychological grounding and validation in AI simulations before real-world use.

Amin Banayeeanzade, Ala N. Tak, Fatemeh Bahrani, Anahita Bolourani, Leonardo Blas, Emilio Ferrara, Jonathan Gratch, Sai Praneeth Karimireddy. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Both haptic signals and simple, non-anthropomorphic robots can convey complex emotions and enhance remote communication. In this study, we integrated a zoomorphic socially expressive Blossom robot and a haptic sleeve to create a novel multimodal telepresence platform for remote social interaction. Through a within-subject user study with 16 participants, we explored the individual and combined effects of socially expressive robots and mediated social touch on affective communication and social presence during a semi-collaborative LEGO assembly task. Across all participants, the robot and wearable device significantly impacted how participants perceived expressions of gratitude, calming, attention-grabbing, and sadness, evaluated through self-reported valence and arousal. The robot and wearable device in our setting did not show a significant effect on social presence. The observations from this exploratory study can inform the design of multimodal telepresence systems and interactions using non-anthropomorphic robots and mediated touch.

Negotiation requires dynamically balancing self-interest and cooperation within the flow of conversation to maximize one's own utility.Yet, existing agents struggle due to bounded rationality in human data, low adaptability to counterpart behavior, and limited strategic reasoning.To address this, we introduce principle-driven negotiation agents, powered by ASTRA, a novel framework for turn-level offer optimization grounded in two core principles: opponent modeling and Tit-for-Tat reciprocity.ASTRA operates in three stages: (1) interpreting counterpart behavior, (2) optimizing counteroffers via a tool-integrated action with a linear programming (LP) solver, and (3) selecting offers based on strategy assessment and the partner's acceptance probability.Through simulations and human evaluations, our agent effectively adapts to an opponent's shifting stance and achieves favorable outcomes through enhanced adaptability and strategic reasoning.Beyond enhancing negotiation performance, it also serves as a powerful coaching tool, offering interpretable strategic feedback and optimal offer recommendations beyond human bounded rationality, with its potential further validated through human evaluation.

Emotion recognition in dynamic social contexts requires an understanding of the complex interaction between facial expressions and situational cues. This paper presents a salience-adjusted framework for context-aware emotion recognition with Bayesian Cue Integration (BCI) and Visual-Language Models (VLMs) to dynamically weight facial and contextual information based on the expressivity of facial cues. We evaluate this approach using human annotations and automatic emotion recognition systems in prisoner’s dilemma scenarios, which are designed to evoke emotional reactions. Our findings demonstrate that incorporating salience adjustment enhances emotion recognition performance, offering promising directions for future research to extend this framework to broader social contexts and multimodal applications.

This study proposes a framework that uses personality prompting with Large Language Models (LLMs) to generate verbal and non-verbal behaviors for virtual agents based on personality traits.Focusing on extraversion, we evaluated the system across two scenarios-negotiation and ice-breaking-using both introverted and extroverted agents.In Experiment 1, we ran agent-agent simulations and conducted linguistic analysis and personality classification to assess whether the LLM-generated language reflected the intended traits, and whether the corresponding nonverbal behaviors differed by personality.In Experiment 2, we conducted a user study to evaluate whether these personality-aligned behaviors were consistent with their intended traits and perceptible to human observers.Our results show that LLMs can generate verbal and nonverbal behaviors that align with personality traits, and that users are able to recognize these traits through the agents' behaviors.This work highlights the potential of LLMs in shaping personality-aligned virtual agents.

James Anthony Hale, Sushrita Rakshit, Kushal Chawla, Jeanne M Brett, Jonathan Gratch. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Abstract Prior work suggests automated dispute resolution tools using “provably fair” algorithms can address disparities between demographic groups. These methods use multi-criteria elicited preferences from all disputants and satisfy constraints to generate “fair” solutions. However, we analyze the potential for inequity to permeate proposals through the preference elicitation stage. This possibility arises if differences in dispositional attitudes differ between demographics, and those dispositions affect elicited preferences. Specifically, risk aversion plays a prominent role in predicting preferences. Risk aversion predicts a weaker relative preference for salary and a softer within-issue utility for each issue; this leads to worse compensation packages for risk-averse groups. These results raise important questions in AI-value alignment about whether an AI mediator should take explicit preferences at face value.

This paper reports two studies investigating the emotional reasoning of Large Language Models (LLM). Previous research has suggested that LLMs are surprisingly accurate at predicting human emotions from text descriptions of situations and reason in a way that is consistent with appraisal theory—a leading theory of emotion. Study 1 tests this claim with a large multilingual corpus (English, French, and German) of autobiographical descriptions of emotionally charged events. We confirm that GPT-4, one of the most advanced and widely studied LLMs, shows a remarkable ability to predict emotion and appraisals. We further show this ability is language-independent, with accuracy being consistent across languages and unaffected by the language of the prompt. However, GPT-4 struggles to accurately predict certain emotions (shame, fear, and irritation) and fails to understand appraisal dimensions related to control and power. We repeat the experiments with Gemini-2.0-Flash and find a remarkably similar pattern of strengths and weaknesses, although it consistently outperforms GPT-4. Study 2 examines a possible mechanism for these failures based on the idea of cognitive appraisal bias. In psychological appraisal theory, appraisal bias is the idea that people evaluate situations in biased, often unrealistic ways. By testing both models on a set of situations designed to identify appraisal bias, we find they exhibit strong—but similar—appraisal bias; for example, evaluating situations as if they were a person high in agreeableness and low in power. We further offer evidence suggesting that LLMs could be debiased by incorporating a person's personality in the prompt. This research underscores LLMs' capabilities and limitations in emotional reasoning, though highlights one mechanism underlying this limitation and suggests an approach for addressing these limits.

As avatars have evolved from simple digital representations into extensions of our identities, they offer unprecedented opportunities for self-expression and customization beyond the physical world limitations. While virtual platforms foster new forms of identity exploration, social norms still play a crucial role in defining what is considered appropriate in these environments. In this study, we surveyed 150 participants to investigate social norms surrounding avatar modifications, examining how perspectives, contexts, and personality traits influence attitudes toward appropriateness. Our findings reveal that avatar modifications are generally viewed as more appropriate when considered from a partner's perspective, especially for changeable attributes. However, these modifications are perceived as less acceptable in professional settings such as workplaces. Additionally, individuals with high self-monitoring tendencies tend to be more resistant to changes, while those scoring higher on Machiavellianism are more accepting of changes, particularly regarding unchangeable attributes and emotional expressions. These findings provide valuable insights for platform developers and designers, highlighting the importance of implementing context-aware customization options that balance core identity elements with personality-driven preferences, thereby enhancing user experiences while respecting social norms.

Emotion recognition in social situations is a complex task that requires integrating information from both facial expressions and the situational context. While traditional approaches to automatic emotion recognition have focused on decontextualized signals, recent research emphasizes the importance of context in shaping emotion perceptions. This paper contributes to the emerging field of context-based emotion recognition by leveraging psychological theories of human emotion perception to inform the design of automated methods. We propose an approach that combines emotion recognition methods with Bayesian Cue Integration (BCI) to integrate emotion inferences from decontextualized facial expressions and contextual knowledge inferred via Large-language Models. We test this approach in the context of interpreting facial expressions during a social task, the prisoner's dilemma. Our results provide clear support for BCI across a range of automatic emotion recognition methods. The best automated method achieved results comparable to human observers, suggesting the potential for this approach to advance the field of affective computing.

People often anthropomorphize agents and show social concern for the agents' goals. Whereas this can be useful to build human-agent cooperation in some settings, in others it can be counterproductive - e.g., when people risk themselves to help a robot. A mechanism, thus, is needed to regulate how much cooperation people show towards agents, according to the context. Here, we show that emotion expressions can be a powerful mechanism to help people identify the appropriate level of cooperation given the situation. In the present study, participants (n=379) engaged in a 20-round iterated prisoner's dilemma game with agents that showed emotional expressions that reflected a preference for maximizing its own interests versus maximizing the participants' interests. Accordingly, the results showed that participants focused significantly more on their own interests when facing the agent that expressed emotions favoring the participants' outcome; moreover, this treatment was more successful in steering the participants' focus to their own interests than showing no emotion. These findings reveal that, in addition to helping build cooperation, as shown in prior work, emotion expression can play a central role in mitigating some negative consequences of anthropomorphizing agents.

Emotional expressions serve as important communicative tools in human negotiations, and prior work has shown that artificial agents can use synthetic expressions to enhance negotiation outcomes and to train negotiation skills. These prior findings have focused on virtual agents and little is known about the effect of expressions when negotiating with physical robots. Therefore, in this study, we compared how participants negotiated with emotionally expressive virtual agents and android robots. Participants$(\mathrm{n}={82})$, as a proposer, played a nonverbal version of a four-issue ultimatum bargaining game with a counterpart who was either a virtual agent or an android robot. Before negotiating, participants observed their counterpart's emotional reactions to potential deals. The results showed that participants were better able to estimate the preferences of virtual counterparts compared with robotic counterpart, and thereby achieve better win-win solutions. We find this effect was mediated by uncanniness: participants found the emotional robot to be uncanny, and this undermined their ability to extract information from the robot's expressions. We discuss theoretical mplications for our understanding of human-robot negotiation and practical implications for the design of effective robot negotiators.

This paper extends recent investigations on the emotional reasoning abilities of Large Language Models (LLMs). Current research on LLMs has not directly evaluated the distinction between how LLMs predict the self-attribution of emotions and the perception of others' emotions. We first look at carefully crafted emotion-evoking stimuli, originally designed to find patterns of brain neural activity representing fine-grained inferred emotional attributions of others. We show that GPT-4 is especially accurate in reasoning about such stimuli. This suggests LLMs agree with humans' attributions of others' emotions in stereotypical scenarios remarkably more than self-attributions of emotions in idiosyncratic situations. To further explore this, our second study utilizes a dataset containing annotations from both the author and a third-person perspective. We find that GPT-4's interpretations align more closely with human judgments about the emotions of others than with self-assessments. Notably, conventional computational models of emotion primarily rely on self-reported ground truth as the gold standard. However, an average observer's standpoint, which LLMs appear to have adopted, might be more relevant for many downstream applications, at least in the absence of individual information and adequate safety considerations.

The intelligent virtual agent community often works from the assumption that embodiment confers clear benefits to human-machine interaction. However, embodiment has potential drawbacks in highlighting the salience of social stereotypes such as those around race and gender. Indeed, theories of computer-mediated communication highlight that visual anonymity can sometimes enhance team outcomes. Negotiation is one domain where social perceptions can impact outcomes. For example, research suggests women perform worse in negotiations and find them more aversive, particularly when interacting with men opponents. Research with human participants makes it challenging to unpack whether these negative consequences stem from women’s perceptions of their partner or greater toughness on the part of these men opponents. We use a socially intelligent AI negotiation agent to begin to unpack these processes. We manipulate the perceived toughness of the AI by whether or not it expresses anger — a common tactic to extract concessions. Independently, we manipulate the activation of stereotypes by randomly setting whether the interaction has embodiment (as a male opponent) or has only text (where we obscure gender cues). We find a clear interaction between gender and embodiment. Specifically, women perform worse, and men perform better against an apparently male opponent compared to a disembodied agent – as measured by the subjective value they assign to their outcome. This highlights the potential disadvantages of embodiment in negotiation, though future research must rule out alternative mechanisms that might explain these results.

Hyundong Cho, Shuai Liu, Taiwei Shi, Darpan Jain, Basem Rizk, Yuyang Huang, Zixun Lu, Nuan Wen, Jonathan Gratch, Emilio Ferrara, Jonathan May. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Negotiation is a complex social interaction that encapsulates emotional encounters in human decision-making. Virtual agents that can negotiate with humans by the means of language are useful in pedagogy and conversational AI. To advance the development of such agents, we explore the role of emotion in the prediction of two important subjective goals in a negotiation – outcome satisfaction and partner perception. We devise ways to measure and compare different degrees of emotion expression in negotiation dialogues, consisting ofemoticon,lexical, andcontextualvariables. Through an extensive analysis of a large-scale dataset in chat-based negotiations, we find that incorporating emotion expression explains significantly more variance, above and beyond the demographics and personality traits of the participants. Further, our temporal analysis reveals that emotive information from both early and later stages of the negotiation contributes to this prediction, indicating the need for a continual learning model of capturing emotion for automated agents. Finally, we extend our analysis to another dataset, showing promise that our findings generalize to more complex scenarios. We conclude by discussing our insights, which will be helpful for designing adaptive negotiation agents that interact through realistic communication interfaces.

Stunning advances in machine learning are heralding a new era in sensing, interpreting, simulating and stimulating human emotion. In the human sciences, research is increasingly highlighting the explanatory power of emotions, feelings, and other affective processes to predict how we think and behave. This is beginning to translate into an explosion of applications that can improve human wellbeing including methods to reduce stress and improve emotion regulation skills, techniques to support healthier social media use, pain monitoring in neonates, and decision-support tools that recognize emotional bias.

Individual differences in preferences allow for integrative (win–win) solutions in negotiations. However, reaching an integrative solution is difficult as each party's preferences and limits are private and must be inferred. We hypothesized that teaching people to infer a generative model of how individuals appraise outcomes and express them as emotional expressions, i.e., an appraisal model, contributes to improving the capability of mental state inference and thus facilitates integrative solutions. In the present study, we compared participants' performance in a 4-issue negotiation after training participants to infer appraisal model during three 2-issue negotiations with a visualized appraisal process and text feedback with the performance of those without inference learning. The results showed that training participants to infer appraisal model helped them better estimate their counterpart's preferences but did not lead them to negotiate more integrative solutions.

Within the field of Affective Computing, facial expressions have traditionally been used as a means of inferring valence and emotions. Most studies have focused on interpreting facial expressions as an isolated signal, typically training algorithms with annotators with a $3^{\text {rd }}$-person point of view, often without access to the original context. When the context is provided, recent research highlights that the interpretation annotators assign to facial expressions is sometimes more strongly influenced by the context than the facial expression. But even when the context is provided, annotators are psychologically and physically distant from the original setting that produced the emotion. In this paper, we explore how context and facial expressions shape $2^{\text {nd }}$-person interpretation of facial expressions and compare this to $1^{\text {st }}$-person self-report. Results show that both expression and context contribute to self and other impressions but in different ways. Expressions and context are independent predictors of $1^{\text {st }}$-person judgments but interact to determine $2^{\text {nd }}$. person judgments. In particular, the way players interpret their partner’s facial cues changes dramatically based on what just occurred in the game. We discuss the implication of these findings for automatic emotion recognition methods.

Dyadic synchrony—the temporal coordination of behavior and emotions between individuals—is a key aspect of social interactions. Although synchrony in social interactions has been extensively studied, researchers continue to investigate the underlying factors and processes that contribute to this phenomenon of dyadic synchrony. We argue that synchrony in social interactions could potentially emerge from two sources: (1) people are responding to each other’s expressions or (2) people are responding to a shared task. It is also possible that synchrony results from a combination of both these factors. The present study investigated the sources of dyadic synchrony during an iterated prisoner’s dilemma (IPD) task. We examined two conditions: a still condition, where dyads only saw a still image of their partner, and a video condition, where dyads had real-time visual access to their partner’s facial reactions. We assessed synchrony using Dynamic Time Warping in real dyads and randomly paired dyads. Our findings revealed that synchrony was present in real interactions even without visual cues, suggesting that shared experiences of reacting to a joint outcome contribute to synchrony. The ability to see the partner’s facial reaction in the video condition further enhanced synchrony, particularly when dyad members’ decisions were not aligned. These results underscore the interplay between shared experiences and visual cues in driving dyadic synchrony during interactive social situations, providing valuable insights into the mechanisms underlying social coordination and alignment in cooperative and competitive contexts.

This paper investigates the emotional reasoning abilities of the GPT family of large language models. We advocate a component perspective on evaluation that decomposes models into different aspects of emotional reasoning (appraisal derivation, affect/intensity derivation, and consequent derivation). We report two studies. A correlational study examines how the model reasons about autobiographical memories. An experimental study systematically varies aspects of situations in ways previously shown to impact emotion intensity and coping tendencies. Results demonstrate, even without prompt engineering, GPT predictions closely match human-provided appraisals and emotion labels, though GPT struggled to predict emotion intensity and coping responses. GPT-4 performed best on the first study but performed poorly on the second (though it yielded the best results following minor prompt engineering). The evaluation raises questions about how to utilize the strengths and mitigate the weaknesses of such models, including dealing with variability in responses. More fundamentally, these studies highlight the benefits of the componential perspective on model evaluation.

The lack of contextual information in text data can make the annotation process of text-based emotion classification datasets challenging. As a result, such datasets often contain labels that fail to consider all the relevant emotions in the vocabulary. This misalignment between text inputs and labels can degrade the performance of machine learning models trained on top of them. As re-annotating entire datasets is a costly and time-consuming task that cannot be done at scale, we propose to use the expressive capabilities of large language models to synthesize additional context for input text to increase its alignment with the annotated emotional labels. In this work, we propose a formal definition of textual context to motivate a prompting strategy to enhance such contextual information. We provide both human and empirical evaluation to demonstrate the efficacy of the enhanced context. Our method improves alignment between inputs and their human-annotated labels from both an empirical and human-evaluated standpoint.

Dialogue systems capable of social influence such as persuasion, negotiation, and therapy, are essential for extending the use of technology to numerous realistic scenarios.However, existing research primarily focuses on either task-oriented or open-domain scenarios, a categorization that has been inadequate for capturing influence skills systematically.There exists no formal definition or category for dialogue systems with these skills and data-driven efforts in this direction are highly limited.In this work, we formally define and introduce the category of social influence dialogue systems that influence users' cognitive and emotional responses, leading to changes in thoughts, opinions, and behaviors through natural conversations.We present a survey of various tasks, datasets, and methods, compiling the progress across seven diverse domains.We discuss the commonalities and differences between the examined systems, identify limitations, and recommend future directions.This study serves as a comprehensive reference for social influence dialogue systems to inspire more dedicated research and discussion in this emerging area.

A natural way to design a negotiation dialogue system is via self-play RL: train an agent that learns to maximize its performance by interacting with a simulated user that has been designed to imitate human-human dialogue data.Although this procedure has been adopted in prior work, we find that it results in a fundamentally flawed system that fails to learn the value of compromise in a negotiation, which can often lead to no agreements (i.e., the partner walking away without a deal), ultimately hurting the model's overall performance.We investigate this observation in the context of DealOrNoDeal task, a multi-issue negotiation over books, hats, and balls.Grounded in negotiation theory from Economics, we modify the training procedure in two novel ways to design agents with diverse personalities and analyze their performance with human partners.We find that although both techniques show promise, a selfish agent, which maximizes its own performance while also avoiding walkaways, performs superior to other variants by implicitly learning to generate value for both itself and the negotiation partner.We discuss the implications of our findings for what it means to be a successful negotiation dialogue system and how these systems should be designed in the future.

Current approaches to building negotiation agents rely either on model-based techniques that explicitly implement key principles of negotiation or model-free techniques leveraging algorithms developed via training on large amounts of human-generated text. We bridge these two approaches by combining a model-based approach with large language models for natural language understanding and generation. We find large language models perform well at recognizing dialogue acts and an opponent's emotions; perform reasonably well at recognizing opponents' preferences in the negotiation; and perform worse at understanding opponent offers. We also perform a qualitative comparison of the capabilities of our hybrid approach with a model-free method and find our hybrid agent provides safeguards against hallucinations and guarantees more control over aspects of negotiation such as emotional expressions, information sharing, and concession strategies.

Virtual humans and social robots frequently generate behaviors that human observers naturally see as expressing emotion. In this review article, we highlight that these expressions can have important benefits for human–machine interaction. We first summarize the psychological findings on how emotional expressions achieve important social functions in human relationships and highlight that artificial emotional expressions can serve analogous functions in human–machine interaction. We then review computational methods for determining what expressions make sense to generate within the context of interaction and how to realize those expressions across multiple modalities, such as facial expressions, voice, language, and touch. The use of synthetic expressions raises a number of ethical concerns, and we conclude with a discussion of principles to achieve the benefits of machine emotion in ethical ways.

For affective computing to have an impact outside the laboratory, facial expressions must be studied in rich naturalistic situations. We argue negotiations are one such situation as they are ubiquitous in daily life, often evoke strong emotions, and perceived emotion shapes decisions and outcomes. Negotiations are a growing focus in AI research and applications, including agents that negotiate directly with people and attempt to use affective information. We introduce the DyNego-WOZ Corpus, which includes dyadic negotiation between participants and wizard-controlled virtual humans. We demonstrate the value of this corpus to the affective computing community by examining participants’ facial expressions in response to a virtual human negotiation partner. We show that people's facial expressions typically co-occur with the end of their partner's speech (suggesting they reflect a reaction to the content of this speech), that these reactions do not correspond to prototypicalemotionalexpressions, and that these reactions can help predict the expresser's subsequent action. We highlight challenges in working with such naturalistic data, including difficulties of expression recognition during speech, and the extreme variability of expressions, both across participants and within a negotiation. Our findings reinforce arguments that facial expressions convey more than emotional state but serve important communicative functions.

We consider the problem of inferring what happened to a person in a social task from momentary facial reactions. To approach this, we introduce several innovations. First, rather than predicting what (observers think) someone feels, we predict objective features of the event that immediately preceded the facial reactions. Second, we draw on appraisal theory, a key psychological theory of emotion, to characterize features of this immediately-preceded event. Specifically, we explore if facial expressions reveal if the event is expected, goal-congruent, and norm-compatible. Finally, we argue thatemotional expressivityserves as a better feature for characterizing momentary expressions than traditional facial features. Specifically, we use supervised machine learning to predict third-party judgments of emotional expressivity with high accuracy, and show this model improves inferences about the nature of the event that preceded an emotional reaction. Contrary to common sense, “genuine smiles” failed to predict if an event advanced a person’s goals. Rather, expressions best revealed if an event violated expectations. We discussed the implications of these findings for the interpretation of facial displays and potential limitations that could impact the generality of these findings.

Much prior work in automated negotiation makes the simplifying assumption of linear utility functions. As such, we propose a framework for multilateral repeated negotiations in a complex game setting---to introduce non-linearities---where negotiators can choose with whom they negotiate in subsequent games. This game setting not only creates non-linear utility functions, but also motivates the negotiation.

Virtual human expressions can shape user behavior [1, 2, 3], yet in negotiation, findings have been underwhelming. For example, human negotiators can use anger to claim value (i.e., extract concessions) [4], but anger has no effect when exhibited by a virtual human [5]. Other psychological work suggests that emotions can create value (e.g., happy negotiators can better discover tradeoffs across issues that "grow the pie"), but little research has examined how virtual human expressions shape value creation. Here we present an agent architecture and pilot study that examines differences between how the emotional expressions of human and virtual-human opponents shape value claiming and value creation. We replicate the finding that virtual human anger fails to influence value claiming but discover counter-intuitive findings on value creation. We argue these findings highlight the potential for intelligent virtual humans to yield insight into human psychology.

Negotiation is a complex social interaction that encapsulates emotional encounters in human decision-making. Virtual agents that can negotiate with humans are useful in pedagogy and conversational AI. To advance the development of such agents, we explore the prediction of two important subjective goals in a negotiation – outcome satisfaction and partner perception. Specifically, we analyze the extent to which emotion attributes extracted from the negotiation help in the prediction, above and beyond the individual difference variables. We focus on a recent dataset in chat-based negotiations, grounded in a realistic camping scenario. We study three degrees of emotion dimensions – emoticons, lexical, and contextual by leveraging affective lexicons and a state-of-the-art deep learning architecture. Our insights will be helpful in designing adaptive negotiation agents that interact through realistic communication interfaces.

Automatic emotion recognition methods are sensitive to the variations across humans and datasets and their performance drops when evaluated across corpora. Domain adaptation (DA) techniques such as Domain-Adversarial Neural Network (DANN) can mitigate this problem. However, domain adaptation cannot guarantee to preserve local features necessary for emotion recognition while reducing domain discrepancies in global features. In this paper, we propose Face wArping emoTion rEcognition (FATE) to address this problem. Unlike the traditional DA models in which the base model is first trained with the source data and then fine-tuned with the source and target data, we reverse the training order. Specifically, we employ first-order facial animation warping to generate a synthetic dataset and utilize contrastive learning to pre-train the encoder. Then, we fine-tune the encoder and the classifier with the source data. After fine-tuning, the model achieves superior emotion recognition performance by preserving the subtle facial features. Our experiments on cross-domain emotion recognition with facial behaviors (Aff-Wild2, SEWA, and SEMAINE) indicate that the proposed FATE model substantially outperforms the domain adaptation models, suggesting that FATE has a better domain generalizability for emotion recognition.

Understanding an opponent's wants is crucial for maximizing the outcomes of a multi-issue negotiation. To do this, automated systems must build an "opponent model" from information conveyed during a negotiation. Bayesian and frequentist models are the most commonly used. Bayesian models have a principled way to incorporate prior knowledge about an opponent's preferences. However, frequentist models have outperformed Bayesian approaches in practice, dominating the yearly agent-verses-agent negotiation competitions. With growing interest in agents that negotiate with people, this presumed dominance needs to be revisited. Human opponents convey far less information than automated agents, and people often share similar preferences (e.g., in a salary negotiation, most people care the most about salary). Thus, the theoretical advantage of Bayesian approaches may translate into practice for agent-versus-human negotiation. In this work, we compare the performance of Bayesian models against a leading frequentist approach in an agent-versus-human multi-issue salary negotiation. Although we show that frequentist opponent models outperform Bayesian models when using a uniform prior, Bayesian approaches excel when using two common priors. The best performance is achieved with an empirically-derived prior (i.e., biasing the model space using the distribution of preferences found in past human negotiators). Yet, strong performance is also observed when using a "fixed-pie bias", the prior used by most human negotiators. We discuss the implication of these findings for research on human-agent negotiation.

Women earn less than men in technical fields. Competing theories have been offered to explain this disparity. Some argue that women underperform in negotiating their salary, in-part due to language in job descriptions, called gender triggers, which leave women feeling disadvantaged in salary negotiations. Others point to structural and institutional bias: i.e., recruiters make better offers to men even when women exhibit equal negotiation skills. As a final salary is co-constructed though an interaction between employees and recruiters, it is difficult to disentangle these views. Here, we discuss how intelligent virtual agents serve as powerful methodological tools that lend new insight into this psychological debate. We use virtual negotiators to examine the impact of gender triggers on computer science (CS) undergraduates that engaged in a simulated salary negotiation with an automated recruiter. We find that, regardless of gender, CS students are reluctant to negotiate, and this hesitancy likely lowers their starting salary. Even when they negotiate, students show little skill in discovering tradeoffs that could enhance their salary, highlighting the need for negotiation training in technical fields. Most importantly, we find little evidence that gender triggers impact women's negotiated outcomes, at least within the field of CS. We argue that findings that emphasize women's individual deficits may reflect a lack of experimental control, which intelligent agents can help correct, and that structural and institutional explanations of inequity deserve greater attention.

Negotiation is a social interaction aimed at reaching a mutually beneficial agreement among all participants in a conflict situation. Unfortunately, parties often find negotiations threatening or aversive, undermining the chances of reaching good agreements. Politeness strategies are means of communicating one's demands to a counterpart without threatening the counterpart's "face" by using tactical phrasing. Politeness strategies are classified into positive, negative, and off-record strategies depending on how they avoid face-threatening acts. In the present study, we investigated whether differences in the politeness strategies used by a virtual agent impact negotiated outcomes in a non-zero-sum situation. The participants (n=106) engaged in an online multi-issue negotiation with one of three agents (using the positive, off-record, or no politeness strategies, while the negative strategy was excluded because of validation failure). The results showed that the agents who used the off-record strategy were able to extract greater concessions from their human partners, whereas positive politeness, which does not threaten the other's face, led to fairer negotiated agreements. The human participants were comfortable exploiting agents who failed to adopt any politeness in their language. Politeness is a part of the toolbox that people use to manage the social rewards and punishments associated with all interactions, and our work highlights that agents can use this important social tool.

Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, Jonathan Gratch. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

An extensive body of research has examined how specific emotional expressions shape social perceptions and social decisions, yet recent scholarship in emotion research has raised questions about the validity of emotion as a construct. In this article, we contrast the value of measuring emotional expressions with the more general construct of expressivity (in the sense of conveying a thought or emotion through any nonverbal behavior) and develop models that can automatically extract perceived expressivity from videos. Although less extensive, a solid body of research has shown expressivity to be an important element when studying interpersonal perception, particularly in psychiatric contexts. Here we examine the role expressivity plays in predicting social perceptions and decisions in the context of a social dilemma. We show that perceivers use more than facial expressions when making judgments of expressivity and see these expressions as conveying thoughts as well as emotions (although facial expressions and emotional attributions explain most of the variance in these judgments). We next show that expressivity can be predicted with high accuracy using Lasso and random forests. Our analysis shows that features related to motion dynamics are particularly important for modeling these judgments. We also show that learned models of expressivity have value in recognizing important aspects of a social situation. First, we revisit a previously published finding which showed that smile intensity was associated with the unexpectedness of outcomes in social dilemmas; instead, we show that expressivity is a better predictor (and explanation) of this finding. Second, we provide preliminary evidence that expressivity is useful for identifying “moments of interest” in a video sequence.

Spatial abilities have been shown to have high predictability in students' success in STEM related fields. Studies have also shown that there is a correlation between students' spatial skills and programming abilities, but it is unknown how well students' prior spatial abilities can predict students' introductory programming abilities at the end of the semester. During this study we used a multinomal logistic regression to create a predictive model to predict students' introductory programming abilities at the end of the semester. The highest model accuracy (64.6%) was obtained when accounting for students' prior programming abilities, prior spatial skills, socioeconomic status, and three factors regarding students' attitudes towards computing. It was also found that when looking at the predictability of each individual variable, students' prior spatial ability had the highest predictability (56.6% accuracy) when compared to all other variables.

Intelligent virtual agents have been developed to study, assess and teach a variety of human interpersonal skills. Here we examine the impact of an agent's perspective-taking sophistication on human negotiators. Good perspective-takers can discover creative solutions that benefit both parties, but many have difficulty with this skill. In particular, novices focus on explicit goal-statements (e.g., "I want apples more than bananas") but discount goal-relevant information implicit in the opponent's offers. Many human-agent negotiation agents similarly ignore implicit information. We examined the influence of implicit information on human negotiators by independently enhancing agents in two ways: do agents communicate implicit information and do they attend to implicit information communicated by users. We find that communicating implicit information seems to confuse user's perspective-taking ability, yet paradoxically, helps lead them to better outcomes. In contrast, an agent that attends to user's implicit communications shows better perspective-taking but fails to translate this into better outcomes. These results emphasize the challenges associated with implicit information. We discuss how these results impact the design of negotiation agents for applications, analysis and pedagogy.

Agents that interact with humans in complex, social tasks need the ability to comprehend as well as employ common social strategies. In negotiation, there is ample evidence of such techniques being used efficaciously in human interchanges. In this work, we demonstrate a new design for socially aware agents that employ one such technique---favor exchange---in order to gain value when playing against humans. In an online study of a robust, simulated social negotiation task, we show that these agents are effective against real human participants. In particular, we show that agents that ask for favors during the course of a repeated set of negotiations are more successful than those that do not. Additionally, previous work has demonstrated that humans can detect when agents betray them by failing to return favors that were previously promised. By contrast, this work indicates that these betrayal techniques may go largely undetected in complex scenarios.

This paper discusses the validation of a modified computer science attitudes instrument. Dorn and Tew's Computing Attitude Survey was modified by adding questions on gender issues and questions regarding students' perceptions of the utility of computing. Current trends indicate an increasing gap in the genders graduating with degrees in computer science. The new questions explore student attitudes and perceptions of women in computing while items related to computer science utility explore the importance of CS in students' lives and careers. These modifications necessitated the re-validation of the new revised instrument and comparison of the results obtained with in the original instrument by Dorn and Tew.

This paper discusses the results of replicating and extending a study performed by Cooper et al. examining the relationship between students' spatial skills and their success in learning to program. Whereas Cooper et al. worked with high school students participating in a summer program, we worked with college students taking an introductory computing course. Like Cooper et al.'s study, we saw a correlation between a student's spatial skills and their success in learning computing. More significantly, we saw that after applying an intervention to teach spatial skills, students demonstrated improved performance both on a standard spatial skills assessment as well as on a CS content instrument. We also saw a correlation between students' enjoyment in computing and improved performance both on a standard spatial skills assessment and on a CS content instrument, a result not observed by Cooper et al.

In social decision-making tasks, facial expressions are informative signals that indicate motives and intentions. As people are aware that their expressions influence partner behavior, expressions may be strategically regulated in competitive environments to influence a social partner's decision-making. In this work, we examine facial expressions and their strategic regulation within the context of an iterated prisoner's dilemma. Utilizing video-cued rating procedures, we examine several key questions about the functionality of facial expressions in social decision-making. First, we assess the extent to which emotion and expression regulation are accurately detected from dynamic facial expressions in interpersonal interactions. Second, we explore which facial cues are utilized to evaluate emotion and regulation information. Finally, we investigate the role of context in participants' emotion and regulation judgments. Results show that participants accurately perceive facial emotion and expression regulation, although they are better at recognizing emotions than regulation. Using automated expression analysis and stepwise regression, we constructed models that use action units from participant videos to predict their video-cued emotion and regulation ratings. We show that these models perform similarly and, in some cases, better than participants do. Moreover, these models demonstrate that game state information improves predictive accuracy, thus implying that context information is important in the evaluation of facial expressions.

This study examines spontaneous facial expressions in an iterated prisoner's dilemma with financial stakes. Our goal was to identify typical facial expressions associated with key events during the interaction (e.g., cooperation or exploitation) and contrast these reactions with alternative theories of the meaning of facial expressions. Specifically, we examined if expressions reflect individual self-interest (e.g., winning) or social motives (e.g., promoting fairness) and the extent to which surprise might moderate the intensity of facial displays. In contrast to predictions of scientific and folk theories of expression, smiles were the only expressions consistently elicited, regardless of the reward or fairness of outcomes. Further, these smiles serve as a reliable indicator of the surprisingness of the event, but not its pleasure (contradicting research on both the meaning of smiles and indicators of surprise). To our knowledge, this is the first study to indicate that smiles signal surprise.

We present the results of the 2ndAnnual Human-Agent League of the Automated Negotiating Agent Competition. Building on the success of the previous year's results, a new challenge was issued that focused exploring the likeability-success tradeoff in negotiations. By examining a series of repeated negotiations, actions may affect the relationship between automated negotiating agents and their human competitors over time. The results presented herein support a more complex view of human-agent negotiation and capture of integrative potential (win-win solutions). We show that, although likeability is generally seen as a tradeoff to winning, agents are able to remain well-liked while winning if integrative potential is not discovered in a given negotiation. The results indicate that the top-performing agent in this competition took advantage of this loophole by engaging in favor exchange across negotiations (cross-game logrolling). These exploratory results provide information about the effects of different submitted “black-box” agents in human-agent negotiation and provide a state-of-the-art benchmark for human-agent design.

Designers of human-agent systems often assume that users interact with machines as if they are interacting with another person. As a consequences, fidelity to human behavior is often viewed as the gold standard for judging agent design, and theories of human social psychology are often accepted without question as a framework for informing human-agent interaction. This assumption was given strength by the pioneering work of Cliff Nass showing that many of the effects studied within social psychology seem to apply to human-machine interaction. In this talk, I will illustrate that these social effects are much weaker than widely supposed, and that the differences in how people treat machines are arguably more interesting than the similarities. These differences can lead to novel insights into human social cognition and unique technological solutions to intractable social problems. I will discuss this in the context of our research on education and mental health. Thus, rather copying human behavior, I will argue that HAI researchers should aim to transcend conventional forms of social interaction, and work towards novel theoretical frameworks that address the novel psychology of human-agent interaction.

Self-disclosure to others has a proven benefit for one’s mental health. It is shown that disclosure to computers can be similarly beneficial for emotional and psychological well-being. In this paper, we analyzed verbal and nonverbal behavior associated with self-disclosure in two datasets containing structured human-human and human-agent interviews from more than 200 participants. Correlation analysis of verbal and nonverbal behavior revealed that linguistic features such as affective and cognitive content in verbal behavior, and nonverbal behavior such as head gestures are associated with intimate self-disclosure. A multimodal deep neural network was developed to automatically estimate the level of intimate self-disclosure from verbal and nonverbal behavior. Between modalities, verbal behavior was the best modality for estimating self-disclosure within-corpora achieving r = 0.66. However, the cross-corpus evaluation demonstrated that nonverbal behavior can outperform language modality in cross-corpus evaluation. Such automatic models can be deployed in interactive virtual agents or social robots to evaluate rapport and guide their conversational strategy.

Research has shown that virtual agents can be effective tools for teaching negotiation. Virtual agents provide an opportuni-ty for students to practice their negotiation skills which leads to better outcomes. However, these negotiation training agents often lack the ability to understand the errors students make when negotiating, thus limiting their effectiveness as training tools. In this article, we argue that automated opponent-modeling techniques serve as effective methods for diagnos-ing important negotiation mistakes. To demonstrate this, we analyze a large number of participant traces generated while negotiating with a set of automated opponents. We show that negotiators' performance is closely tied to their understanding of an opponent's preferences. We further show that opponent modeling techniques can diagnose specific errors includ-ing: failure to elicit diagnostic information from an opponent, failure to utilize the information that was elicited, and failure to understand the transparency of an opponent. These results show that opponent modeling techniques can be effective methods for diagnosing and potentially correcting crucial ne-gotiation errors.

Recent research shows that how we respond to other social actors depends on what sort of mind we ascribe to them. In this article we examine how perceptions of a virtual agent's mind shape behavior in human-agent negotiations. We varied descriptions and communicative behavior of virtual agents on two dimensions according to the mind perception theory:agency (cognitive aptitude) andpatiency (affective aptitude). Participants then engaged in negotiations with the different agents. People scored more points and engaged in shorter negotiations with agents described to be cognitively intelligent, and got lower points and had longer negotiations with agents that were described to be cognitively unintelligent. Accordingly, agents described as having low agency ended up earning more points than those with high agency. Within the negotiations themselves, participants sent more happy and surprise emojis and emotionally valenced messages to agents described to be emotional. This high degree of described patiency also affected perceptions of the agent's moral standing and relatability. In short, manipulating the perceived mind of agents affects how people negotiate with them. We discuss these results, which show that agents are perceived not only as social actors, but as intentional actors through negotiations.

Research in social science distinguishes between two types of social influence: informational and normative. Informational social influence is driven by the desire to evaluate ambiguous situations correctly, whereas normative social influence is driven by the desire to be liked and gain social acceptance from another person. Although we know from research that humans can effectively use either of these techniques to persuade other humans, scholars have yet to examine the relative effectiveness of informational versus normative social influence when used by virtual agents. We report a study in which users interact with a system that persuades them either using informational or normative social influence. Furthermore, to compare agents to human interlocutors, users are told that the system is either tele-operated by a human (avatar) or fully-automated (agent). Using this design, we are able to compare the effectiveness of virtual agents (vs humans) in employing informational versus normative social influence. Participants interacted with the system, which employed a Wizard-of-Oz operated virtual agent that tried to persuade the user to agree with its rankings on a "survival task." Controlling for initial divergence in rankings between user and the agent, there was a significant main effect such that informational social influence resulted in greater influence than normative influence. However, this was qualified by an interaction that approached significance; users were, if anything, more persuaded by informational influence when they believe the agent was AI (compared to a human), whereas there was no difference between the agent and avatar in the normative influence condition.

We present the results of a machine-learning approach to the analysis of several human-agent negotiation studies. By combining expert knowledge of negotiating behavior compiled over a series of empirical studies with neural networks, we show that a hybrid approach to parameter selection yields promise for designing -more effective and socially intelligent agents. Specifically, we show that a deep feedforward neural network using a theory-driven three-parameter model can be effective in predicting negotiation outcomes. Furthermore, it outperforms other expert-designed models that use more parameters, as well as those using other, more limited techniques (such as linear regression models or boosted decision trees). We anticipate these results will have impact for those seeking to combine extensive domain knowledge with more automated approaches in human-computer negotiation.

Socially intelligent artificial agents and robots are anticipated to become ubiquitous in home, work, and military environments. With the addition of such agents to human teams it is crucial to evaluate their role in the planning, decision making, and conflict mediation processes. We conducted a study to evaluate the utility of a virtual agent that provided mission planning support in a three-person human team during a military strategic mission planning scenario. The team consisted of a human team lead who made the final decisions and three supporting roles, two humans and the artificial agent. The mission outcome was experimentally designed to fail and introduced a conflict between the human team members and the leader. This conflict was mediated by the artificial agent during the debriefing process through discuss or debate and open communication strategies of conflict resolution [1]. Our results showed that our teams experienced conflict. The teams also responded socially to the virtual agent, although they did not find the agent beneficial to the mediation process. Finally, teams collaborated well together and perceived task proficiency increased for team leaders. Socially intelligent agents show potential for conflict mediation, but need careful design and implementation to improve team processes and collaboration.

\beginabstract Cognitive tests have been long used as a measure of student knowledge, ability, and as a predictor for success in engineering and computer science. However, these tests are not without their own problems relating to priming, difficulty (resulting in test fatigue) and time on exam. This paper discusses efforts to modify Parker et al.'s Second CS1 aptitude test (SCS1) \citeParker16 to reduce the time spent on the exam, provide greater customization to match concepts taught across three universities, and reduce redundancy of test questions all while maintaining the instrument's reliability. This instrument was modified for use on an ongoing grant investigating whether spatial abilities impact the success of students in introductory CS courses. The instrument developed in this paper is a revised shortened version of Second Computer Science 1 (SCS1) aptitude test, designated as SCS1R. \endabstract

Behavioral intervention strategies have yet to become successful in the development of initiatives to foster pro-environmental behaviors in buildings. In this paper, we explored the potentials of increasing the effectiveness of requests aiming to promote pro-environmental behaviors by engaging users in a social dialog, given the effects of two possible personas that are more related to the buildings (i.e., building vs. building manager). We tested our hypotheses and evaluated our findings in virtual and physical environments and found similar effects in both environments. Our results showed that social dialog involvement persuaded respondents to perform more pro-environmental actions. However, these effects were significant when the requests were delivered by an agent representing the building. In addition, these strategies were not equally effective across all types of people and their effects varied for people with different characteristics. Our findings provide useful design choices for persuasive technologies aiming to promote pro-environmental behaviors.

A crucial aspect of virtual gaming experiences is the avatar: the player's virtual self-representation. While research has demonstrated benefits to using self-similar avatars in some virtual experiences, such avatars sometimes produce a more negative experience for women. To help researchers and game designers assess the cost-benefit tradeoffs of self-similar avatars, we compared players' performance and subjective experience in a search and rescue computer game when using two different photorealistic avatars: their own self or a friend, and when playing either a social (rescuing people) or a nonsocial (rescuing gems) version of the game. There was no effect of avatar appearance on players' performance or subjective experience in either game version, but we also found that women's experience with self-similar avatars was no more negative than men's. Our results suggest that avatar appearance may not make a difference to players in certain game contexts.

Predicting how a person will respond can be very useful, for instance when designing a strategy for negotiations. We investigate whether it is possible for machine learning and computer vision techniques to recognize a person's intentions and predict their actions based on their visually expressive behaviour, where in this paper we focus on the face. We have chosen as our setting pairs of humans playing a simplified version of poker, where the players are behaving naturally and spontaneously, albeit mediated through a computer connection. In particular, we ask if we can automatically predict whether a player is going to fold or not. We also try to answer the question of at what time point the signal for predicting if a player will fold is strongest. We use state-of-the-art FACS Action Unit detectors to automatically annotate the players facial expressions, which have been recorded on video. In addition, we use timestamps of when the player received their card and when they placed their bets, as well as the amounts they bet. Thus, the system is fully automated. We are able to predict whether a person will fold or not significantly better than chance based solely on their expressive behaviour starting three seconds before they fold.

This work explores the extent to which social dialogue can mitigate (or exacerbate) the loss of trust caused when robots make conversational errors. Our study uses a NAO robot programmed to persuade users to agree with its rankings on two tasks. We perform two manipulations: (1) The timing of conversational errors - the robot exhibited errors either in the first task, the second task, or neither; (2) The presence of social dialogue - between the two tasks, users either engaged in a social dialogue with the robot or completed a control task. We found that the timing of the errors matters: replicating previous research, conversational errors reduce the robot's influence in the second task, but not on the first task. Social dialogue interacts with the timing of errors, acting as an intensifier: social dialogue helps the robot recover from prior errors, and actually boosts subsequent influence; but social dialogue backfires if it is followed by errors, because it extends the period of good performance, creating a stronger contrast effect with the subsequent errors. The design of social robots should therefore be more careful to avoid errors after periods of good performance than early on in a dialogue.

A long tradition of research suggests a relationship between emotional mimicry and pro-social behavior, but the nature of this relationship is unclear. Does mimicry cause rapport and cooperation, or merely reflect it? Virtual humans can provide unique insights into these social processes by allowing unprecedented levels of experimental control. In a 2 x 2 factorial design, we examined the impact of facial mimicry and counter-mimicry in the iterated prisoner's dilemma. Participants played with an agent that copied their smiles and frowns or one that showed the opposite pattern -- i.e., that frowned when they smiled. As people tend to smile more than frown, we independently manipulated the contingency of expressions to ensure any effects are due to mimicry alone, and not the overall positivity/negativity of the agent: i.e., participants saw either a reflection of their own expressions or saw the expressions shown to a previous participant. Results show that participants smiled significantly more when playing an agent that mimicked them. Results also show a complex association between smiling, feelings of rapport, and cooperation. We discuss the implications of these findings on virtual human systems and theories of cooperation.

This work explores whether culture impacts the extent to which social dialogue can mitigate (or exacerbate) the loss of trust caused when agents make conversational errors. Our study uses an agent designed to persuade users to agree with its rankings on two tasks. Participants from the U.S. and Japan completed our study. We perform two manipulations: (1) The presence of conversational errors -- the agent exhibited errors in the second task or not; (2) The presence of social dialogue -- between the two tasks, users either engaged in a social dialogue with the agent or completed a control task. Replicating previous research, conversational errors reduce the agent's influence. However, we found that culture matters: there was a marginally significant three-way interaction with culture, presence of social dialogue, and presence of errors. The pattern of results suggests that, for American participants, social dialogue backfired if it is followed by errors, presumably because it extends the period of good performance, creating a stronger contrast effect with the subsequent errors. However, for Japanese participants, social dialogue if anything mitigates the detrimental effect of errors; the negative effect of errors is only seen in the absence of a social dialogue. Agent design should therefore take the culture of the intended users into consideration when considering use of social dialogue to bolster agents against conversational errors.

Research has demonstrated promising benefits of applying virtual trainers to promote physical fitness. The current study investigated the value of virtual agents in the context of personal fitness, compared to trainers with greater levels of perceived agency (avatar or live human). We also explored the possibility that the effectiveness of the virtual trainer might depend on the affective tone it uses when trying to motivate users. Accordingly, participants received either positively or negatively valenced motivational messages from a virtual human they believed to be either an agent or an avatar, or they received the messages from a human instructor via skype. Both self-report and physiological data were collected. Like in-person coaches, the live human trainer who used negatively valenced messages were well-regarded; however, when the agent or avatar used negatively valenced messages, participants responded more poorly than when they used positively valenced ones. Perceived agency also affected rapport: compared to the agent, users felt more rapport with the live human trainer or the avatar. Regardless of trainer type, they also felt more rapport - and said they put in more effort - with trainers that used positively valenced messages than those that used negatively valenced ones. However, in reality, they put in more physical effort (as measured by heart rate) when trainers employed the more negatively valenced affective tone. We discuss implications for human--computer interaction.

We present the results of the first annual Human-Agent League of ANAC. By introducing a new human-agent negotiating platform to the research community at large, we facilitated new advancements in human-aware agents. This has succeeded in pushing the envelope in agent design, and creating a corpus of useful human-agent interaction data. Our results indicate a variety of agents were submitted, and that their varying strategies had distinct outcomes on many measures of the negotiation. These agents approach the problems endemic to human negotiation, including user modeling, bidding strategy, rapport techniques, and strategic bargaining. Some agents employed advanced tactics in information gathering or emotional displays and gained more points than their opponents, while others were considered more "likeable" by their partners.

We present the results of a study in which humans negotiate with computerized agents employing varied tactics over a repeated number of economic ultimatum games. We report that certain agents are highly effective against particular classes of humans: several individual difference measures for the human participant are shown to be critical in determining which agents will be successful. Asking for favors works when playing with pro-social people but backfires with more selfish individuals. Further, making poor offers invites punishment from Machiavellian individuals. These factors may be learned once and applied over repeated negotiations, which means user modeling techniques that can detect these differences accurately will be more successful than those that don't. Our work additionally shows that a significant benefit of cooperation is also present in repeated games---after sufficient interaction. These results have deep significance to agent designers who wish to design agents that are effective in negotiating with a broad swath of real human opponents. Furthermore, it demonstrates the effectiveness of techniques which can reason about negotiation over time.

The importance of rigorous standards in computer science education research to include a description of hypotheses, research questions, methodologies, and results has been recognized in the computer science education community. The driving force for computer science education research is to understand the learning needs of our students who are human subjects. Therefore, some computer science education researchers may need to answer a critical question before they start their planned research: Do I need Institutional Review Board (IRB) approval to conduct this research using my students as research subjects? The key goal of the IRB is to protect human subjects from physical or psychological harm ("Code of Federal Regulations, Title 45, Public Welfare, Part 46, Protection of Human Subjects"). Although commonly used in the fields of health and social sciences, the role and purpose of IRB, the different categories of IRB reviews, the timeline from planning and submission of an IRB application, and the general rules for citing IRB in publications and grant proposals are not widely understood in the computer science education research community. In this poster, the authors describe the history, the purpose, review categories, and guidelines for reporting on the IRB. The authors will tailor the discussions on the different IRB review categories to computer science educators interested in conducting computer science education research with their students.

research-article Share on ACM Transactions on Interactive Intelligent Systems (TiiS) Special Issue on Trust and Influence in Intelligent Human-Machine Interaction Authors: Benjamin A. Knott The Office of Naval Research Global, Roppongi, Tokyo, Japan The Office of Naval Research Global, Roppongi, Tokyo, JapanView Profile , Jonathan Gratch University of Southern California, USA University of Southern California, USAView Profile , Angelo Cangelosi Plymouth University, USA Plymouth University, USAView Profile , James Caverlee Texas A8M University, USA Texas A8M University, USAView Profile Authors Info & Claims ACM Transactions on Interactive Intelligent SystemsVolume 8Issue 4December 2018 Article No.: 25pp 1–3https://doi.org/10.1145/3281451Published:16 November 2018Publication History 0citation395DownloadsMetricsTotal Citations0Total Downloads395Last 12 Months62Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

No abstract available.

Emotions have a temporal nature and very often personality traits and underlying psychological conditions are hidden in the dynamics of those expressions. Within this work, we investigate the dynamics of the facial displays of dyads during an iterated social dilemma. We focus on the effect of gender and gender-pairing on those behaviors. We use networks to capture the temporal dynamics and create measures of inter- and intra-personal dependencies of emotional states. Our analysis on an iterated prisoner's dilemma corpus suggests that there are gender differences on the transitions of the emotional states and the degree of emotional influence from the opponent.

Many automatic facial expression recognizers now output individual facial action units (AUs), but several lines of evidence suggest that it is the combination of AUs that is psychologically meaningful: e.g., (a) constraints arising from facial morphology, (b) prior published evidence, (c) claims arising from basic emotion theory. We performed factor analysis on a large data set and recovered factors that have been discussed in the literature as psychologically meaningful. Further we show that some of these factors have external validity in that they predict participant behaviors in an iterated prisoner's dilemma task and in fact with more precision than the individual AUs. These results both reinforce the validity of automatic recognition (as these factors would be expected from accurate AU detection) and suggest the benefits of using such factors for understanding these facial expressions as social signals.

Social robots establish rapport with human users. This work explores the extent to which rapport-building can benefit (or harm) conversations with robots, and under what circumstances this occurs. For example, previous work has shown that agents that make conversational errors are less capable of influencing people than agents that do not make errors [1]. Some work has shown this effect with robots, but prior research has not considered additional factors such as the level of rapport between the person and the robot. We predicted that building rapport through a social dialogue (such as an ice-breaker) could mitigate the detrimental effect of a robot's errors on influence. Our study used a Nao robot programmed to persuade users to agree with its rankings on two "survival tasks" (e.g., lunar survival task). We manipulated both errors and social dialogue:the robot either exhibited errors in the second survival task or not, and users either engaged in an ice-breaker with the robot between the two survival tasks or completed a control task. Replicating previous research, errors tended to reduce the robot's influence in the second survival task. Contrary to our prediction, results revealed that the ice-breaker did not mitigate the effect of errors, and if anything, errors were more harmful after the ice-breaker (intended to build rapport) than in the control condition. This backfiring of attempted rapport-building may be due to a contrast effect, suggesting that the design of social robots should avoid introducing dialogues of incongruent quality.

Computers that negotiate on our behalf hold great promise for the future and will even become indispensable in emerging application domains such as the smart grid and the Internet of Things. Much research has thus been expended to create agents that are able to negotiate in an abundance of circumstances. However, up until now, truly autonomous negotiators have rarely been deployed in real-world applications. This paper sizes up current negotiating agents and explores a number of technological, societal and ethical challenges that autonomous negotiation systems have brought about. The questions we address are: in what sense are these systems autonomous, what has been holding back their further proliferation, and is their spread something we should encourage? We relate the automated negotiation research agenda to dimensions of autonomy and distill three major themes that we believe will propel autonomous negotiation forward: accurate representation, long-term perspective, and user trust. We argue these orthogonal research directions need to be aligned and advanced in unison to sustain tangible progress in the field.

The seventh Audio-Visual Emotion Challenge and workshop AVEC 2017 was held in conjunction with ACM Multimedia'17. This year, the AVEC series addresses two distinct sub-challenges: emotion recognition and depression detection. The Affect Sub-Challenge is based on a novel dataset of human-human interactions recorded 'in-the-wild', whereas the Depression Sub-Challenge is based on the same dataset as the one used in AVEC 2016, with human-agent interactions. In this summary, we mainly describe participation and its conditions.

We demonstrate two agents, a robot and a virtual human, which can be used for studying factors that impact social influence. The agents engage in dialogue scenarios that build familiarity, share information, and attempt to influence a human participant. The scenarios are variants of the classical “survival task,” where members of a team rank the importance of a number of items (e.g., items that might help one survive a crash in the desert). These are ranked individually and then re-ranked following a team discussion, and the difference in ranking provides an objective measure of social influence. Survival tasks have been used in psychology, virtual human research, and human-robot interaction. Our agents are operated in a “Wizard-of-Oz” fashion, where a hidden human operator chooses the agents’ dialogue actions while interacting with an experiment participant.

This paper builds on prior psychological studies that identify signals of trustworthiness between two human negotiators. Unlike prior work, the current work tracks such signals automatically and fuses them into computational models that predict trustworthiness. To achieve this goal, we apply automatic trackers to recordings of human dyads negotiating in a multi-issue bargaining task. We identify behavioral indicators in different modalities (facial expressions, gestures, gaze, and conversational features) that are predictive of trustworthiness. We predict both objective trustworthiness (i.e., are they honest) and perceived trustworthiness (i.e., do they seem honest to their interaction partner). Our experiments show that people are poor judges of objective trustworthiness (i.e., objective and perceived trustworthiness are predicted by different indicators), and that multimodal approaches better predict objective trustworthiness, whereas people overly rely on facial expressions when judging the honesty of their partner. Moreover, domain knowledge (from the literature and prior analysis of behaviors) facilitates the model development process.

Recent advances in scanning technology have enabled the widespread capture of 3D character models based on human subjects. Intuition suggests that, with these new capabilities to create avatars that look like their users, every player should have his or her own avatar to play video games or simulations. We explicitly test the impact of having one's own avatar (vs. a yoked control avatar) in a simulation (i.e., maze running task with mines). We test the impact of avatar identity on both subjective (e.g., feeling connected and engaged, liking avatar's appearance, feeling upset when avatar's injured, enjoying the game) and behavioral variables (e.g., time to complete task, speed, number of mines triggered, riskiness of maze path chosen). Results indicate that having an avatar that looks like the user improves their subjective experience, but there is no significant effect on how users perform in the simulation.

The sixth Audio-Visual Emotion Challenge and workshop AVEC 2016 was held in conjunction ACM Multimedia'16. This year the AVEC series addresses two distinct sub-challenges, multi-modal emotion recognition and audio-visual depression detection. Both sub-challenges are in a way a return to AVEC's past editions: the emotion sub-challenge is based on the same dataset as the one used in AVEC 2015, and depression analysis was previously addressed in AVEC 2013/2014. In this summary, we mainly describe participation and its conditions.

Reduced frequency range in vowel production is a well documented speech characteristic of individuals with psychological and neurological disorders. Affective disorders such as depression and post-traumatic stress disorder (PTSD) are known to influence motor control and in particular speech production. The assessment and documentation of reduced vowel space and reduced expressivity often either rely on subjective assessments or on analysis of speech under constrained laboratory conditions (e.g. sustained vowel production, reading tasks). These constraints render the analysis of such measures expensive and impractical. Within this work, we investigate an automatic unsupervised machine learning based approach to assess a speaker's vowel space. Our experiments are based on recordings of 253 individuals. Symptoms of depression and PTSD are assessed using standard self-assessment questionnaires and their cut-off scores. The experiments show a significantly reduced vowel space in subjects that scored positively on the questionnaires. We show the measure's statistical robustness against varying demographics of individuals and articulation rate. The reduced vowel space for subjects with symptoms of depression can be explained by the common condition of psychomotor retardation influencing articulation and motor control. These findings could potentially support treatment of affective disorders, like depression and PTSD in the future.

Guilt and envy play an important role in social interaction. Guilt occurs when individuals cause harm to others or break social norms. Envy occurs when individuals compare themselves unfavorably to others and desire to benefit from the others’ advantage. In both cases, these emotions motivate people to act and change the status quo: following guilt, people try to make amends for the perceived transgression, and following envy, people try to harm envied others. In this article, we present two experiments that study participants’ experience of guilt and envy when engaging in social decision making with machines and humans. The results showed that, though experiencing the same level of envy, people felt considerably less guilt with machines than with humans. These effects occurred both with subjective and behavioral measures of guilt and envy, and in three different economic games: public goods, ultimatum, and dictator game. This poses an important challenge for human-computer interaction because, as shown here, it leads people to systematically exploit machines, when compared to humans. We discuss theoretical and practical implications for the design of human-machine interaction systems that hope to achieve the kind of efficiency -- cooperation, fairness, reciprocity, etc. -- we see in human-human interaction.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

Appraisal theory is the most influential theory within affective computing, and serves as the basis for several computational models of emotion. The theory makes strong claims of domain-independence: seemingly different situations, both within and across domains are claimed to produce the identical emotional responses if and only if they are appraised the same way. This article tests this claim, and the predictions of a computational model that embodies it, in two very different interactive games. The results extend prior empirical evidence for appraisal theory to situations where emotions unfold and change over time.

Sporting events evoke strong emotions amongst fans and thus act as natural laboratories to explore emotions and how they unfold in the wild. Computational tools, such as sentiment analysis, provide new ways to examine such dynamic emotional processes. In this article we use sentiment analysis to examine tweets posted during 2014 World Cup. Such analysis gives insight into how people respond to highly emotional events, and how these emotions are shaped by contextual factors, such as prior expectations, and how these emotions change as events unfold over time. Here we report on some preliminary analysis of a World Cup twitter corpus using sentiment analysis techniques. We show these tools can give new insights into existing theories of what makes a sporting match exciting. This analysis seems to suggest that, contrary to assumptions in sports economics, excitement relates to expressions of negative emotion. We also discuss some challenges that such data present for existing sentiment analysis techniques and discuss future analysis.

Even with the rise in use of TeleMedicine for health care and mental health, research suggests that clinicians may have difficulty reading nonverbal cues in computer-mediated situations. However, the recent progress in tracking affective markers (i.e., displays of emotional expressions on face and in voice) has opened the door to new clinical applications that might help health care providers better read nonverbal behaviors when employing TeleMedicine. For example, an interface that automatically quantified affective markers could assist clinicians in their assessment of and treatment for psychological distress (i.e., symptoms of depression and Post-traumatic Stress Disorder (PTSD)). To move towards this prospect, we will show that clinicians' judgments of these nonverbal affective markers (e.g., smile, frown, eye contact, tense voice) could be informed by such technology. The results of our evaluation suggest that clinicians' ratings of nonverbal affective markers are less predictive of psychological distress than automatically quantified affective markers. Because such quantifications are more strongly associated with psychological distress than clinician ratings of these same nonverbal behaviors, an affective interface providing quantifications of nonverbal affective markers could potentially improve assessment of psychological distress..

Negotiation between virtual agents and humans is a complex field that requires designers of systems to be aware not only of the efficient solutions to a given game, but also the mechanisms by which humans create value over multiple negotiations. One way of considering the agent's impact beyond a single negotiation session is by considering the use of external “ledgers” across multiple sessions. We present results that describe the effects of favor exchange on negotiation outcomes, fairness, and trust for two distinct cross-cultural populations, and illustrate the ramifications of their similarities and differences on virtual agent design.

Research shows that people consistently reach more efficient solutions than those predicted by standard economic models, which assume people are selfish. Artificial intelligence, in turn, seeks to create machines that can achieve these levels of efficiency in human-machine interaction. However, as reinforced in this paper, people's decisions are systematically less efficient - i.e., less fair and favorable - with machines than with humans. To understand the cause of this bias, we resort to a well-known experimental economics model: Fehr and Schmidt's inequity aversion model. This model accounts for people's aversion to disadvantageous outcome inequality (envy) and aversion to advantageous outcome inequality (guilt). We present an experiment where participants engaged in the ultimatum and dictator games with human or machine counterparts. By fitting this data to Fehr and Schmidt's model, we show that people acted as if they were just as envious of humans as of machines; but, in contrast, people showed less guilt when making unfavorable decisions to machines. This result, thus, provides critical insight into this bias people show, in economic settings, in favor of humans. We discuss implications for the design of machines that engage in social decision making with humans.

Machiavellianism, by definition, is the tendency to use other people as a tool to achieve one's own goals. Despite the large focus on the Big Five traits of personality, this anti-social trait is relatively unexplored in the computational realm. Automatically recognizing anti-social traits can have important uses across a variety of applications. In this paper, we use negotiation as a setting that provides Machiavellians with the opportunity to reveal their exploitative inclinations. We use textual, visual, acoustic, and behavioral cues to automatically predict High vs. Low Machiavellian personalities. These learned models have good accuracy when compared with other personality-recognition methods, and we provide evidence that the automatically-learned models are consistent with existing literature on this anti-social trait, giving evidence that these results can generalize to other domains.

Emotional signaling plays an important role in negotiations and other social decision-making tasks as it can signal intention and shape joint decisions. Specifically it has been shown to influence cooperation or competition. This has been shown in previous studies for scripted interactions that control emotion signaling and rely on manual coding of affect. In this work we examine face-to-face interactions in an iterative social dilemma task (prisoner's dilemma) via an automatic framework for facial expression analysis. We explore if automatic analysis of emotion can give insight into the social function of emotion in face-to-face interactions. Our analysis suggests that positive and negative displays of emotion are associated with more prosocial and proself game acts respectively. Moreover signaling cooperative intentions to the opponent via positivity can leave participants more open to exploitation, whereas signaling a more tough stance via negativity seems to discourage exploitation. However, the benefit of negative affect is short-term and both players do worse over time if they show negative emotions.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. With this demo, we focus our attention on the perception part of the system, a multimodal framework which captures and analyzes user state behavior for both behavioral understanding and interactional purposes. We will demonstrate real-time user state sensing as a part of the SimSensei architecture and discuss how this technology enabled automatic analysis of behaviors related to psychological distress.

Reduced frequency range in vowel production is a well documented speech characteristic of individuals' with psychological and neurological disorders. Depression is known to influence motor control and in particular speech production. The assessment and documentation of reduced vowel space and associated perceived hypoarticulation and reduced expressivity often rely on subjective assessments. Within this work, we investigate an automatic unsupervised machine learning approach to assess a speaker's vowel space within three distinct speech corpora and compare observed vowel space measures of subjects with and without psychological conditions associated with psychological distress, namely depression, post-traumatic stress disorder (PTSD), and suicidality. Our experiments are based on recordings of over 300 individuals. The experiments show a significantly reduced vowel space in conversational speech for depression, PTSD, and suicidality. We further observe a similar trend of reduced vowel space for read speech. A possible explanation for a reduced vowel space is psychomotor retardation, a common symptom of depression that influences motor control and speech production.

Recent research in perception and theory of mind reveals that people show different behavior and lower activation of brain regions associated with mentalizing (i.e., the inference of other's mental states) when engaged in decision making with computers, when compared to humans. These findings are important for affective computing because they suggest people's decisions might be influenced differently according to whether they believe emotional expressions shown in computers are being generated by algorithms or humans. To test this, we had people engage in a social dilemma (Experiment 1) or negotiation (Experiment 2) with virtual humans that were either perceived to be agents (i.e., controlled by computers) or avatars (i.e., controlled by humans). The results showed that such perceptions have a deep impact on people's decisions: in Experiment 1, people cooperated more with virtual humans that showed cooperative facial displays (e.g., joy after mutual cooperation) than competitive displays (e.g., joy when the participant was exploited) but, the effect was stronger with avatars (d = .601) than with agents (d = .360); in Experiment 2, people conceded more to angry than neutral virtual humans but, again, the effect was much stronger with avatars (d = 1.162) than with agents (d = .066). Participants also showed less anger towards avatars and formed more positive impressions of avatars when compared to agents.

Negotiation is a component deeply ingrained in our daily lives, and it can be challenging for a person to predict the respondent's reaction (acceptance or rejection) to a negotiation offer. In this work, we focus on finding acoustic and visual behavioral cues that are predictive of the respondent's immediate reactions using a face-to-face negotiation dataset, which consists of 42 dyadic interactions in a simulated negotiation setting. We show our results of exploring four different sources of information, namely nonverbal behavior of the proposer, that of the respondent, mutual behavior between the interactants related to behavioral symmetry and asymmetry, and past negotiation history between the interactants. Firstly, we show that considering other sources of information (other than the nonverbal behavior of the respondent) can also have comparable performance in predicting respondent reactions. Secondly, we show that automatically extracted mutual behavioral cues of symmetry and asymmetry are predictive partially due to their capturing information of the nature of the interaction itself, whether it is cooperative or competitive. Lastly, we identify audio-visual behavioral cues that are most predictive of the respondent's immediate reactions.

Agency - the capacity to plan and act - and experience - the capacity to sense and feel - are two critical aspects that determine whether people will perceive non-human entities, such as autonomous agents, to have a mind. There is evidence that the absence of either can reduce cooperation. We present an experiment that tests the necessity of both for cooperation with agents. In this experiment we manipulated people's perceptions about the cognitive and affective abilities of agents, when engaging in the ultimatum game. The results indicated that people offered more money to agents that were perceived to make decisions according to their intentions (high agency), rather than randomly (low agency). Additionally, the results showed that people offered more money to agents that expressed emotion (high experience), when compared to agents that did not (low experience). We discuss the implications of this agency-experience theoretical framework for the design of artificially intelligent decision makers.

Transactions on Affective Computing (TAC) reaches its fifth anniversary I have many positive developments to note. First, and most significant for our readers, TAC's first official impact factors were released and we came in at a whopping 3.47. This is extraordinary for a new journal and makes us the second highest rated journal in IEEE's Computer Society. For those interested in artificial intelligence research, this places us well above the top-ranked Artificial Intelligence (2.71). It even places us above the top journal for emotion research, Emotion (3.37). Clearly the field of affective computing has grown up.

Recent research in neuroeconomics reveals that people show different behavior and lower activation of brain regions associated with mentalizing (i.e., the inference of other's mental states) when engaged in decision making tasks with a computer, when compared to a human. These findings are important for affective computing because they suggest people's decision making might be influenced differently according to whether they believe the emotional expressions shown by a computer are being generated by a computer algorithm or a human. To test this, we had people engage in a social dilemma (Experiment 1) or a negotiation (Experiment 2) with virtual humans that were either agents (i.e., controlled by computers) or avatars (i.e., controlled by humans). The results show a clear agency effect: in Experiment 1, people cooperated more with virtual humans that showed facial cooperative displays (e.g., joy after mutual cooperation) rather than competitive displays (e.g., joy when the participant was exploited) but, the effect was only significant with avatars, in Experiment 2, people conceded more to an angry than a neutral virtual human but, once again, the effect was only significant with avatars.

In this paper, we analyze face-to-face negotiation interactions with the goal of predicting the respondent's immediate reaction (i.e., accept or reject) to a negotiation offer. Supported by the theory of social rapport, we focus on mutual behaviors which are defined as nonverbal characteristics that occur due to interactional influence. These patterns include behavioral symmetry (e.g., synchronized smiles) as well as asymmetry (e.g., opposite postures) between the two negotiators. In addition, we put emphasis on finding audio-visual mutual behaviors that can be extracted automatically, with the vision of a real-time decision support tool. We introduce a dyadic negotiation dataset consisting of 42 face-to-face interactions and show experiments confirming the importance of multimodal and mutual behaviors.

In this paper, we show that gender plays an important role in the automatic assessment of psychological conditions such as depression and post-traumatic stress disorder (PTSD). We identify a directly interpretable and intuitive set of predictive indicators, selected from three general categories of nonverbal behaviors: affect, expression variability and motor variability. For the analysis, we introduce a semi-structured virtual human interview dataset which includes 53 video recorded interactions. Our experiments on automatic classification of psychological conditions show that a gender-dependent approach significantly improves the performance over a gender agnostic one.

This paper presents a brief summary of the first workshop on Emotion Representation and Modelling in Human-Computer-Interaction-Systems. The ERM4HCI 2013 workshop is held in conjunction with the ICMI 2013 conference. The focus is on theory driven representation and modelling of emotions in the context of Human-Computer-Interaction.

Negotiations can be characterized by the strategy participants adopt to achieve their ends (e.g., individualistic strategies are based on self-interest, cooperative strategies are used when participants try to maximize the joint gain, while competitive strategies focus on maximizing each participant’s score against the other) and the outcomes that each participant achieves in the negotiation. This paper investigates the process and the result of predicting the outcome and strategy of participants throughout the progress of the negotiation by using basic, easy to extract, linguistic and acoustic features. We evaluate our approach on a face-to-face negotiation dataset consisting of 41 dyadic interactions and show that it’s possible to significantly improve over a majority-class baseline in tasks of predicting the strategy and outcome of the interaction by analyzing only basic low level features of the negotiation.

We seek to investigate voice quality characteristics, in particular on a breathy to tense dimension, as an indicator for psychological distress, i.e. depression and post-traumatic stress disorder (PTSD), within semi-structured virtual human interviews. Our evaluation identifies significant differences between the voice quality of psychologically distressed participants and not-distressed participants within this limited corpus. We investigate the capability of automatic algorithms to classify psychologically distressed speech in speaker-independent experiments. Additionally, we examine the impact of the posed questions’ affective polarity, as motivated by findings in the literature on positive stimulus attenuation and negative stimulus potentiation in emotional reactivity of psychologically distressed participants. The experiments yield promising results using standard machine learning algorithms and solely four distinct features capturing the tenseness of the speaker’s voice.

With this fourth year of the IEEE Transactions on Affective Computing (TAC), the field of affective computing is strong and vibrant. In 2013 we will mark the fifth International Conferences on Affective Computing in Geneva, Switzerland, which will be cochaired by our associate editor, Catherine Pelachaud. Over the last year, TAC published 43 articles over four issues, up from 26 articles in 2011. We expect to maintain this publication rate over the next year and focus on attracting high-quality articles. The journal continues to encourage interdisciplinary research and we've attracted articles from recognized names in both the computational and social sciences of affect. After these four years of growth, as Editor-in-Chief (EiC) I feel confident in stating that TAC is the premier journal for research on the topic of affective computing. The editorial board has remained steady over the last year, but to handle our increasing paper load we have added one editor focusing on human-robot interaction. I welcome Bilge Mutlu from the University of Wisconsin, Madison, who's bio and photo are provided. In the coming year, my primary goal continues to be to increase the visibility of the journal and for this I need your help. Please help me in spreading awareness of the journal.

The past years have seen increasing cooperation between psychology and computer science in the field of computational modeling of emotion. However, to realize its potential, the exchange between the two disciplines, as well as the intradisciplinary coordination, should be further improved. We make three proposals for how this could be achieved. The proposals refer to: 1) systematizing and classifying the assumptions of psychological emotion theories; 2) formalizing emotion theories in implementation-independent formal languages (set theory, agent logics); and 3) modeling emotions using general cognitive architectures (such as Soar and ACT-R), general agent architectures (such as the BDI architecture) or general-purpose affective agent architectures. These proposals share two overarching themes. The first is a proposal for modularization: deconstruct emotion theories into basic assumptions; modularize architectures. The second is a proposal for unification and standardization: Translate different emotion theories into a common informal conceptual system or a formal language, or implement them in a common architecture.

Be it in our workplace or with our family or friends, negotiation comprises a fundamental fabric of our everyday life, and it is apparent that a system that can automatically predict negotiation outcomes will have substantial implications. In this paper, we focus on finding nonverbal behaviors that are predictive of immediate outcomes (acceptances or rejections of proposals) in a dyadic negotiation. Looking at the nonverbal behaviors of the respondent alone would be inadequate since ample predictive information could also reside in the behaviors of the proposer, as well as the past history between the two parties. With this intuition in mind, we show that a more accurate prediction can be achieved by considering all the three sources (multimodal) of information together. We evaluate our approach on a face-to-face negotiation dataset consisting of 42 dyadic interactions and show that integrating all three sources of information outperforms each individual predictor.

Social causality is the inference an entity makes about the social behavior of other entities and self. Besides physical cause and effect, social causality involves reasoning about epistemic states of agents and coercive circumstances. Based on such inference, responsibility judgment is the process whereby one singles out individuals to assign responsibility, credit or blame for multi-agent activities. Social causality and responsibility judgment are a key aspect of social intelligence, and a model for them facilitates the design and development of a variety of multi-agent interactive systems. Based on psychological attribution theory, this paper presents a domain-independent computational model to automate social inference and judgment process according to an agent’s causal knowledge and observations of interaction. We conduct experimental studies to empirically validate the computational model. The experimental results show that our model predicts human judgments of social attributions and makes inferences consistent with what most people do in their judgments. Therefore, the proposed model can be generically incorporated into an intelligent system to augment its social and cognitive functionality.

ABSTRACT Previous research illustrates that people can be influenced by the emotional displays of computer‐generated agents. What is less clear is if these influences arise from cognitive or affective process (i.e., do people use agent displays as information or do they provoke user emotions). To unpack these processes, we examine the decisions and physiological reactions of participants (heart rate and electrodermal activity) when engaged in a decision task (prisoner's dilemma game) with emotionally expressive agents. Our results replicate findings that people's decisions are influenced by such emotional displays, but these influences differ depending on the extent to which these displays provoke an affective response. Specifically, we show that an individual difference known as electrodermal lability predicts the extent to whether people will engage affectively or strategically with such agents, thereby better predicting their decisions. We discuss implications for designing agent facial expressions to enhance social interaction between humans and agents. Copyright © 2012 John Wiley & Sons, Ltd.

When interacting with computer agents, people make inferences about various characteristics of these agents, such as their reliability and trustworthiness. These perceptions are significant, as they influence people's behavior towards the agents, and may foster or inhibit repeated interactions between them. In this paper we investigate whether computer agents can use the expression of emotion to influence human perceptions of trustworthiness. In particular, we study human-computer interactions within the context of a negotiation game, in which players make alternating offers to decide on how to divide a set of resources. A series of negotiation games between a human and several agents is then followed by a "trust game." In this game people have to choose one among several agents to interact with, as well as how much of their resources they will trust to it. Our results indicate that, among those agents that displayed emotion, those whose expression was in accord with their actions (strategy) during the negotiation game were generally preferred as partners in the trust game over those whose emotion expressions and actions did not mesh. Moreover, we observed that when emotion does not carry useful new information, it fails to strongly influence human decision-making behavior in a negotiation setting.

Communication is more effective and persuasive when par-ticipants establish rapport. Tickle-Degnen and Rosenthal [57] argue rapport arises when participants exhibit mutual attentiveness, positivity and coordination. In this paper, we investigate how these factors relate to perceptions of rap-port when users interact via avatars in virtual worlds. In this study, participants told a story to what they believed was the avatar of another participant. In fact, the avatar was a computer program that systematically manipulated levels of attentiveness, positivity and coordination. In contrast to Tickel-Degnen and Rosenthal’s findings, high-levels of mutual attentiveness alone can dramatically lower percep-tions of rapport in avatar communication. Indeed, an agent that attempted to maximize mutual attention performed as poorly as an agent that was designed to convey boredom. Adding positivity and coordination to mutual attentiveness, on the other hand, greatly improved rapport. This work un-veils the dependencies between components of rapport and informs the design of agents and avatars in computer me-diated communication. Author Keywords Virtual human, rapport, back-channel, gaze, head nod,

Abstract We explored the relationship between interactants' social anxiety and the interactional fidelity of virtual humans. We specifically addressed whether the contingent non‐verbal feedback of virtual humans affects the association between interactants' social anxiety and their verbal self‐disclosure. This subject was investigated across three experimental conditions where participants interacted with real human videos and virtual humans in computer‐mediated interview interactions. The results demonstrated that socially anxious people revealed more information and greater intimate information about themselves when interacting with a virtual human when compared with real human video interaction, whereas less socially anxious people did not show this difference. We discuss the implication of this association between the interactional fidelity of virtual humans and social anxiety in a human interactant on the design of an embodied virtual agent for social skills' training and psychotherapy. Copyright © 2010 John Wiley & Sons, Ltd.

Abstract Affect has been shown to influence respiration in people. This paper takes this insight and proposes a real‐time model to express affect through respiration in virtual humans. Fourteen affective states are explored: excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy. Specific respiratory patterns are described from the literature for each of these affective states. Then, a real‐time model of respiration is proposed that uses morphing to animate breathing and provides parameters to control respiration rate, respiration depth and the respiration cycle curve. These parameters are used to implement the respiratory patterns. Finally, a within‐subjects study is described where subjects are asked to classify videos of the virtual human expressing each affective state with or without the specific respiratory patterns. The study was presented to 41 subjects and the results show that the model improved perception of excitement, pain, relief, boredom, anger, fear, panic, disgust, and startle. Copyright © 2010 John Wiley & Sons, Ltd.

Research show that teacher's nonverbal immediacy can have a positive impact on student's cognitive learning and affect [3]. This paper investigates the effectiveness of nonverbal immediacy using a virtual human. The virtual human attempts to use immediacy feedback to create rapport with the learner. Results show that the virtual human established rapport with learners but did not help them achieve better learning results. The results also suggest that creating rapport is related to higher self-efficacy, and self-efficacy is related to better learning results.

'Works of art (…) can be expressive of human qualities: one of the most characteristic and pervasive features of art is that percepts (lines, colors, progressions of musical tones) can be and are suffused with affect.'

During face-to-face conversation, people use visual feedback such as head nods to communicate relevant information and to synchronize rhythm between participants. In this paper we describe how contextual information from other participants can be used to predict visual feedback and improve recognition of head gestures in human-human interactions. For example, in a dyadic interaction, the speaker contextual cues such as gaze shifts or changes in prosody will influence listener backchannel feedback (e.g., head nod). To automatically learn how to integrate this contextual information into the listener gesture recognition framework, this paper addresses two main challenges: optimal feature representation using an encoding dictionary and automatic selection of optimal feature-encoding pairs. Multimodal integration between context and visual observations is performed using a discriminative sequential model (Latent-Dynamic Conditional Random Fields) trained on previous interactions. In our experiments involving 38 storytelling dyads, our context-based recognizer significantly improved head gesture recognition performance over a vision-only recognizer.

The Institute for Creative Technologies is a federally funded research center set up three years ago at the University of Southern California to advance the state of the art in immersive training. Teaming researchers in artificial intelligence, graphics, animation and immersive audio with Hollywood writers, directors and special effect artists, the ICT brings a unique mix of high-technology and professional storytelling esthetic to the problem of creating compelling immersive environments. This afternoon tutorial will consist of a panel presentation by top ICT affiliated researchers to discuss this wide range of technologies and skills and how they relate to the design of virtual environments. The panel will be followed by a tour of the ICT facilities and demonstrations of several virtual training systems.

I describe a collaborative effort between members of the entertainment and research communities to advance the state of immersive training technology. Pulling together Hollywood's expertise in story, visual effects and production, with expertise in graphics, gaming, artificial intelligence, linguistics, cognitive science, and audio processing, a group of researchers, storytellers and graphic artists is trying to approximate the Holodeck (the ultimate training and entertainment device of the 24th century, as popularized by the TV series Star Trek).

Although most scheduling problems are NP-hard, domain specific techniques perform well in practice but are quite expensive to construct. In adaptive problem-solving solving, domain specific knowledge is acquired automatically for a general problem solver with a flexible control architecture. In this approach, a learning system explores a space of possible heuristic methods for one well-suited to the eccentricities of the given domain and problem distribution. In this article, we discuss an application of the approach to scheduling satellite communications. Using problem distributions based on actual mission requirements, our approach identifies strategies that not only decrease the amount of CPU time required to produce schedules, but also increase the percentage of problems that are solvable within computational resource limitations.

This paper considers the decision-making problem of selecting a strategy from a set of alternatives on the basis of incomplete information (e.g. a finite number of observations). At any time the system can adopt a particular strategy or decide to gather additional information at some cost. Balancing the expected utility of the new information against the cost of acquiring the information is the central problem that the authors address. In the authors' approach, the cost and utility of applying a particular strategy to a given problem are represented as random variables from a parametric distribution. By observing the performance of each strategy on a randomly selected sample of problems, one can use parameter estimation techniques to infer statistical models of performance on the general population of problems. These models can then be used to estimate: (1) the utility and cost of acquiring additional information and (2) the desirability of selecting a particular strategy from a set of choices. Empirical results are presented that demonstrate the effectiveness of the hypothesis evaluation techniques for tuning system parameters in a NASA antenna scheduling application.>