Virtual training environments experienced with different immersive technologies can accommodate users' preferences, proficiency, and platform availability. Whereas research comparing the effects of immersive technologies can provide important insights about their impact on users' experience (e.g. engagement, transfer of learning), current studies do not address how to design the user interface (UI) to ensure sound comparisons across platforms. For effective comparisons, however, the UI designs must be adapted for the platform used to provide comparable usability. In this article we describe our UI design methodology for the development of an effective and usable virtual classroom training simulator built for three technologies: (1) desktop; (2) Head-Mounted Display (HMD); and (3) Cave Automatic Virtual Environment (CAVE). Usability and other user experience factors were evaluated for each platform with concurrent think-aloud protocol and semi-structured interviews indicating that all three UIs were easy to use and to learn. We discuss insights for future development of cross-platform VTEs.

Elementary school classrooms are emotionally stressful environments, for both students and teachers. Successful teachers use strategies that regulate students' emotions and behaviors while also controlling their own emotions (stress, nervousness). To prepare teachers for the challenges of teaching, teacher training should include emotional and behavioral management strategies. Virtual Training Environments (VTEs) are effective at providing experiences and increasing learning in many domains. Creating VTEs for teachers can improve student learning and teacher retention. We introduce our current research aimed at integrating emotionally-intelligent virtual students within a 3D classroom training system. In our simulation, virtual students' emotional states will be determined from an appraisal process of actions taken by the teacher trainee in the virtual classroom. Virtual students will then display the appropriate non-verbal behaviors and react to the teacher accordingly. We present the first steps required to implement our proposed architecture which are based on appraisal theory of emotions and emotion regulation theory.

Simulation-based training systems have proven effective in a variety of domains, both for facilitating the learning of skills as well for applying this knowledge to real life. Although difficulties managing students' disruptive behavior in classrooms has been identified as one of the main causes of teachers' turnover, only a handful of virtual training environments have focused on providing training to teachers, and still no clear methodologies exist for their design, their implementation, nor their evaluation.

Although socially interactive agents have emerged as a new metaphor for human-computer interaction, they are, to date, absent from the Internet. We describe the design choices, implementation, and challenges in building EEVA, the first fully integrated platform-independent framework for deploying realistic 3D web-based social agents: with real-time multimodal perception of, and response to, the user's verbal and non-verbal social cues, EEVA agents are capable of communicating rich customizable content to users in real time, while building and maintaining users' profiles for long-term interactions. The modularity of the EEVA framework enables it to be used as a testbed for agents' social communication model development of increasing performance and sophistication (e.g. building rapport, expressing empathy). We discuss the framework's feasibility by analyzing the response time of the system over the Internet, in the context of a health intervention built using EEVA authoring functionalities.

OBJECTIVE: This research focuses on the potential ability of animated avatars (a digital representation of the user) and virtual agents (a digital representation of a coach, buddy, or teacher) to deliver computer-based interventions for adolescents' chronic weight management. An exploration of the acceptance and desire of teens to interact with avatars and virtual agents for self-management and behavioral modification was undertaken. MATERIALS AND METHODS: The utilized approach was inspired by community-based participatory research. Data was collected from 2 phases: Phase 1) focus groups with teens, provider interviews, parent interviews; and Phase 2) mid-range prototype assessment by teens and providers. RESULTS: Data from all stakeholder groups expressed great interest in avatars and virtual agents assisting self-management efforts. Adolescents felt the avatars and virtual agents could: 1) reinforce guidance and support, 2) fit within their lifestyle, and 3) help set future goals, particularly after witnessing the effect of their current behavior(s) on the projected physical appearance (external and internal organs) of avatars. Teens wanted 2 virtual characters: a virtual agent to act as a coach or teacher and an avatar (extension of themselves) to serve as a "buddy" for empathic support and guidance and as a surrogate for rewards. Preferred modalities for use include both mobile devices to accommodate access and desktop to accommodate preferences for maximum screen real estate to support virtualization of functions that are more contemplative and complex (e.g., goal setting). Adolescents expressed a desire for limited co-user access, which they could regulate. Data revealed certain barriers and facilitators that could affect adoption and use. DISCUSSION: The current study extends the support of teens, parents, and providers for adding avatars or virtual agents to traditional computer-based interactions. Data supports the desire for a personal relationship with a virtual character in support of previous studies. The study provides a foundation for further work in the area of avatar-driven motivational interviewing. CONCLUSIONS: This study provides evidence supporting the use of avatars and virtual agents, designed using participatory approaches, to be included in the continuum of care. Increased probability of engagement and long-term retention of overweight, obese adolescent users and suggests expanding current chronic care models toward more comprehensive, socio-technical representations.

With the growing number of researchers interested in modeling the inner workings of affective social intelligence, the need for tools to easily model its associated expressions has emerged. The goal of this article is two-fold: 1) we describe HapFACS, a free software and API that we developed to provide the affective computing community with a resource that produces static and dynamic facial expressions for three-dimensional speaking characters; and 2) we discuss results of multiple experiments that we conducted in order to scientifically validate our facial expressions and head animations in terms of the widely accepted Facial Action Coding System (FACS) standard, and its Action Units (AU). The result is that users, without any 3D-modeling nor computer graphics expertise, can animate speaking virtual characters with FACS-based realistic facial expression animations, and embed these expressive characters in their own application(s). The HapFACS software and API can also be used for generating repertoires of realistic FACS-validated facial expressions, useful for testing emotion expression generation theories.

The authors first give a brief overview of how computers can afford multiple forms of transformational experiences. Some of these experiences can be purposely designed, for example, to detect and regulate students' affective states to improve aspects of their learning experiences. They can also be used in computer-based psychological interventions that treat psychological illness or that preventively promote wellbeing, healthy lifestyles, and mental health. This special section aims to contribute ideas, methods and case studies for how affective computing can move toward the goal of promoting wellbeing. The papers spread across some of the multiple modalities and techniques used to perceive and detect affect from text, physiology facial expressions, and gaze, and some of the approaches to analyze the expression of sentiments shared via online communities by depressed patients. The articles in this special section discuss how information that computers collect about our behaviour, cognition - and particularly affect - can be used in the further understanding, nurturing or development of wellbeing and human strengths: e.g. self-understanding, empathy, intrinsic motivation toward wellbeing healthy lifestyles.

We present HapFACS (ver. beta), a new open source software and API for generating FACS-based facial expressions on 3D virtual characters that have accompanying lip-synchronized animation abilities. HapFACS has two main usage scenarios: First, with the HapFACS software, users can generate repertoires of realistic FACS-validated facial expressions, either as static images or as videos, Second, with the accessible HapFACS API, users can animate speaking virtual characters with real-time realistic facial expressions, and embed these expressive characters in their own application(s) without any prior experience in computer graphics and modeling. We describe how HapFACS (1) provides control over 49 FACS Action Units at all levels of intensity, (2) enables the animation of faces with a single AU or a composition of AUs, activated unilaterally or bilaterally, and (3) can be applied to any supported character in the underlying 3D-character system1. Finally, we provide details of evaluation experiments we conducted with FACS-certified scorers to validate the facial expressions generated by HapFACS.

This special section is focused on the novel and practical ways, and solid contributions, to improve affective computing and pervasive technologies. There are 19 papers in this special section.

During last decade, an increasing interest for interpreting users' emotional subjective experience on the basis of physiological signals has led to various approaches. In this article, we focus on two different approaches toward emotion recognition : (1) the user-dependency of psycho-physiological data collected (i.e. do we choose to keep track of the specificity of individuals' responses or do we ignore such specificity), and (2) the degree of subjectivity of the stimuli used to elicit emotions (i.e. stimuli with high level of agreement in terms of what emotional experience they elicit among a population can be chosen versus stimuli without such an agreement). In order to assess the implications of adopting one of these methodologies on the personalization of emotion recognition from physiological signals we present our empirical results for emotion recognition from physiological signals based on an experiment involving 40 subjects. We conclude by proposing requirements for any chosen approach to achieve suitable online emotion recognition, in an out-of-the-lab context (e.g. interactive art, e-Health application).

In this article, we discuss the strong relationship between affect and cognition and the importance of emotions in multimodal human computer interaction (HCI) and user-modeling. We introduce the overall paradigm for our multi-modal system that aims at recognizing its users' emotions and at responding to them accordingly depending upon the current context or application. We then describe the design of the emotion elicitation experiment we conducted by collecting, via wearable computers, physiological signals from the autonomic nervous system (galvanic skin response, heart rate, temperature) and mapping them to certain emotions (sadness, anger, fear, surprise, frustration, and amusement). We show the results of three different supervised learning algorithms that categorize these collected signals in terms of emotions, and generalize their learning to recognize emotions from new collections of signals. We finally discuss possible broader impact and possible applications of emotion recognition for multimodal intelligent systems

Near to real-time emotion recognition is a promising task for human-computer interaction (HCI) and human-robot interaction (HRI). Using knowledge about the user's emotions depends upon the possibility to extract information about users' emotions during HCI or HRI without explicitly asking users about the feelings they are experiencing. To be able to sense the user's emotions without interrupting the HCI, we present a new method applied to the emotional experience of the user for extracting semantic information from the autonomic nervous system (ANS) signals associated with emotions. We use the concepts of 1stperson - where the subject consciously (and subjectively) extracts the semantic meaning of a given lived experience, (e.g. `I felt amused') - and 3rdperson approach - where the experimenter interprets the semantic meaning of the subject's experience from a set of externally (and objectively) measured variables (e.g. galvanic skin response measures). Based on the 3rdperson approach, our technique aims at psychologically interpreting physiological parameters (skin conductance and heart rate), and at producing a continuous extraction of the user's affective state during HCI or HRI. We also combine it with the 1stperson approach measure which allows a tailored interpretation of the physiological measure closely related to the user own emotional experience

The development of an autonomous social robot, Cherry, is occurring in tandem with studies gaining potential user preferences, likes, dislikes, and perceptions of her features. Thus far, results have indicated that individuals 1) believe that service robots with emotion and personality capabilities would make them more acceptable in everyday roles in human life, 2) prefer that robots communicate via both human-like facial expressions, voice, and text-based media, 3) become more positive about the idea of service and social robots after exposure to the technology, and 4) find the appearance and facial features of Cherry pleasing. The results of these studies provide the basis for future research efforts, which are discussed.

Human intelligence is being increasingly redefined to include the all-encompassing effect of emotions upon what used to be considered 'pure reason'. With the recent progress of research in computer vision, speech/prosody recognition, and bio-feedback, real-time recognition of affect will enhance human-computer interaction considerably, as well as assist further progress in the development of new emotion theories.In this article, we describe how affect, moods and emotions closely interact with cognition and how affect and emotion are the quintessential multimodal processes in humans. We then propose an adaptive system architecture designed to sense the user's emotional and affective states via three multimodal subsystems (V, K, A): namely (1) the Visual (from facial images and videos), (2) Kinesthetic (from autonomic nervous system (ANS) signals), and (3) Auditory (from speech). The results of the system sensing are then integrated into the multimodal perceived multimodal anthropomorphic interface agent then adapts its interface by responding most appropriately to the current emotional states of its user, and provides intelligent multi-modal feedback to the user.

Article Share on A technical demonstration of Lola, the robot entertainer Authors: Andreas H. Marpaung University of Central Florida, Orlando, FL University of Central Florida, Orlando, FLView Profile , Sarah M. Brown University of Central Florida, Orlando, FL University of Central Florida, Orlando, FLSearch about this author , Christine L. Lisetti University of Central Florida, Orlando, FL University of Central Florida, Orlando, FLView Profile Authors Info & Claims MULTIMEDIA '02: Proceedings of the tenth ACM international conference on MultimediaDecember 2002Pages 91–93https://doi.org/10.1145/641007.641027Published:01 December 2002Publication History 1citation200DownloadsMetricsTotal Citations1Total Downloads200Last 12 Months1Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In this paper, we uncover a new potential application for multimedia technologies: car interfaces for enhanced driver's safety. We also describe the experiment we conducted in order to map certain physiological signals (galvanic skin response, heart beat, and temperature) to certain emotions (Neutral, Anger, Fear, Sadness, and Frustration). We demonstrate the results we gained and describe how we use these results to our Multimodal Affective Driver Interface for the drivers of the future cars.

Previous experiences show that it is possible for agents such as robots cooperating asynchronously on a sequential task to enter deadlock, where one robot does not fulfil its obligations in a timely manner due to hardware or planning failure, unanticipated delays, etc. Our approach uses a formal multilevel hierarchy of emotions where emotions both modify active behaviors at the sensory-motor level and change the set of active behaviors at the schematic level. The resulting implementation of a team of heterogeneous robots using a hybrid deliberative/reactive architecture produced the desired emergent societal behavior. Data collected at two different public venues illustrate how a dependent agent selects new behaviors (e.g., stop serving, move to intercept the refiner) to compensate for delays from a subordinate agent (e.g., blocked by the audience). The subordinate also modifies the intensity of its active behaviors in response to feedback from the dependent agent. The agents communicate asynchronously through knowledge query and manipulation language via wireless Ethernet.

Article Free Access Share on Panel on affect and emotion in the user interface Authors: Barbara Hayes-Roth Stanford University, Gates Computer Science Building, Stanford, CA Stanford University, Gates Computer Science Building, Stanford, CAView Profile , Gene Ball Microsoft Research, One Microsoft Way, Redmond, WA Microsoft Research, One Microsoft Way, Redmond, WAView Profile , Christine Lisetti Stanford University, Department of Computer Science and Department of Psychology, Stanford, CA Stanford University, Department of Computer Science and Department of Psychology, Stanford, CAView Profile , Rosalind W. Picard MIT Media Laboratory, E15-392, 20 Ames Street, Cambridge, MA MIT Media Laboratory, E15-392, 20 Ames Street, Cambridge, MAView Profile , Andrew Stern PF. Magic, 501 2nd St., Suite 400, San Francisco, CA PF. Magic, 501 2nd St., Suite 400, San Francisco, CAView Profile Authors Info & Claims IUI '98: Proceedings of the 3rd international conference on Intelligent user interfacesJanuary 1998 Pages 91–94https://doi.org/10.1145/268389.268406Online:01 January 1998Publication History 16citation1,051DownloadsMetricsTotal Citations16Total Downloads1,051Last 12 Months29Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF