Our research studies early face-body perception of socially anxious individuals during social interaction with a virtual agent (VA) in VR using EEG. VAs’ expressiveness is manipulated during social interactions through their facial animations portraying realistic positive/negative/neutral expressions. Facial expressions of the VAs are recorded using real-time facial animation performance capture and pre-validated. Wearing an HMD, tactile-based VR controllers and a mobile EEG system, participants will interact with individual VAs in a virtual office setting (an employee meeting their employer with a handshake), a scenario known to generate anxiety. Behavioural, physiological, and EEG data will be analysed to reveal the effect of emotional valence on early face-body perception during social interactions. This study provides a framework for synchronised multimodal data recording and analysis.

In this paper we present details of a virtual tour and game for VR headset that are designed to investigate an interactive and engaging approach of applying VR to student recruitment for an undergraduate course. The VR tour employs a floating menu to navigate through a set of 360° panoramic photographs of the teaching environment and uses hotspot interaction to display further information about the course. The VR game is a fast-paced shooting game. The course information is embedded on cubes that the player needs to focus on and destroy. The game experience is expected to generate an engaging way to promote the course. This work in progress outlines the concept and development of the prototype, and discusses the next stages of testing in order to evaluate the effectiveness of applying VR to undergraduate student recruitment.

In this paper, we present an approach for generating dialogues for characters within the context of computational narratives \nusing personality-based features for deep neural networks. The approach integrates the requirements of both narrative genres and personality traits for the definition of character-based stylistic models. \nThe modelling of characters’ features from existing datasets of complete stories permits the generation of personality-rich character dialogues. We present early results from an evaluation based on a sample of characters’ personality traits across different narrative genres, \ndemonstrating variability in the resulting dialogues

The visualization of procedural knowledge from textual documents using 3D animation may be a way to improve understanding. We are interested in applying this approach to documents relating to patient education for bariatric surgery: a domain with challenging textual documents describing behavior recommendations that contain few procedural steps and leave much commonsense knowledge unspecified. In this work we look at how to automatically capture knowledge from a range of differently phrased recommendations and use that with implicit knowledge about compliance and violation, such that the recommendations can be visualized using 3D animations. Our solution is an end-to-end system that automates this process via: analysis of input recommendations to uncover their conditional structure; the use of commonsense knowledge and deontic logic to generate compliance and violation rules; and mapping of this knowledge to update a default knowledge base, which is used to generate appropriate sequences of visualizations. In this paper we overview this approach and demonstrate its potential.

Despite recent progress in plan-based narrative generation, one major limitation is that systems tend to produce a single plotline whose progression entirely determines the narrative experience. However, for certain narrative genres such as serial dramas and soaps, multiple interleaved subplots are expected by the audience, as this tends to be the norm in real-world, human-authored narratives. Current narrative generation techniques have overlooked this important requirement, something which could improve the perceived quality of generated stories. To this end, we have developed a flexible plan-based approach to multiplot narrative generation, that successfully generates narratives conforming to different subplot profiles, in terms of the number of subplots interleaved and the relative time spent on each presentation. We have identified specific challenges such as: distribution of virtual characters across subplots; length of each subplot presentation; and transitioning between subplots.

Narrative generation represents an application domain for AI planning where plan quality is related to properties such as shape of plan trajectory. In our work we have developed a plan-based approach to narrative generation that uses character relationships as a key determinant in controlling plan shape (relationships are key in genres such as serial dramas and soaps). Our approach is implemented in a demonstration Interactive Narrative, called NetworkING, set in the medical drama genre. The system features a user-friendly mechanism for specifying relationships between virtual characters, via a social network and real-time visualisation of generated narratives on a 3D stage.

User interaction with Embodied Conversational Agents (ECA) should involve a significant affective component to achieve realism in communication. This aspect has been studied through different frameworks describing the relationship between user and ECA, for instance alignment, rapport and empathy. We conducted an experiment to explore how an ECA's non-verbal expression can be controlled to respond to a single affective dimension generated by users as input. Our system is based on the mapping of a high-level affective dimension, approach/avoidance, onto a new ECA control mechanism in which Action Units (AU) are activated through a neural network. Since 'approach' has been associated to prefrontal cortex activation, we use a measure of prefrontal cortex left-asymmetry through fNIRS as a single input signal representing the user's attitude towards the ECA. We carried out the experiment with 10 subjects, who have been instructed to express a positive mental attitude towards the ECA. In return, the ECA facial expression would reflect the perceived attitude under a neurofeedback paradigm. Our results suggest that users are able to successfully interact with the ECA and perceive its response as consistent and realistic, both in terms of ECA responsiveness and in terms of relevance of facial expressions. From a system perspective, the empirical calibration of the network supports a progressive recruitment of various AUs, which provides a principled description of the ECA response and its intensity. Our findings suggest that complex ECA facial expressions can be successfully aligned with one high-level affective dimension. Furthermore, this use of a single dimension as input could support experiments in the fine-tuning of AU activation or their personalization to user preferred modalities.

Neurofeedback systems can be modeled as closed-loop control systems with negative feedback. However, little work to date has investigated the potential of this representation in gaining a better understanding of the actual dynamics of neurofeedback towards explaining subjects' performance. In this paper, we analyze neurofeedback training data through a PID control model. We first show that PID model fitting can produce curves that are qualitatively aligned to the measured BCI signal. Secondly, we examine how brain activity during neurofeedback can be related to common characteristics of control systems. For this, we formalized a pre-existing neurofeedback EEG experiment using a Simulink®model that captures both the neural activity and the external algorithm that was utilized to generate the feedback signal. We then used a regression model to fit individual trial data to PID coefficients for the control model. Our results suggest that successful trials tend to be associated to higher average values of Ki, which represents the error-reducing component of the PID controller. It hints that convergence in successful neurofeedback is progressive but complete in approaching the target.

Functional near-infrared spectroscopy (fNIRS) holds increasing potential for Brain-Computer Interfaces (BCI) due to its portability, ease of application, robustness to movement artifacts, and relatively low cost. The use of fNIRS to support the development of affective BCI has received comparatively less attention, despite the role played by the prefrontal cortex in affective control, and the appropriateness of fNIRS to measure prefrontal activity. We present an active, fNIRS-based neurofeedback (NF) interface, which uses differential changes in oxygenation between the left and right sides of the dorsolateral prefrontal cortex to operationalize BCI input. The system is activated by users generating a state of anger, which has been previously linked to increased left prefrontal asymmetry. We have incorporated this NF interface into an experimental platform adapted from a virtual 3D narrative, in which users can express anger at a virtual character perceived as evil, causing the character to disappear progressively. Eleven subjects used the system and were able to successfully perform NF despite minimal training. Extensive analysis confirms that success was associated with the intent to express anger. This has positive implications for the design of affective BCI based on prefrontal asymmetry.

Previous Interactive Storytelling systems have been designed to allow active user intervention in an unfolding story, using established multi-modal interactive techniques to influence narrative development. In this paper we instead explore the use of a form of passive interaction where users' affective responses, measured by physiological proxies, drive a process of narrative adaptation. We introduce a system that implements a passive interaction loop as part of narrative generation, monitoring users' physiological responses to an on-going narrative visualization and using these to adapt the subsequent development of character relationships, narrative focus and pacing. Idiomatic cinematographic techniques applied to the visualization utilize existing theories of establishing characteristic emotional tone and viewer expectations to foster additional user response. Experimental results support the applicability of filmic emotional theories in a non-film visual realization, demonstrating significant appropriate user physiological response to narrative events and "emotional cues". The subsequent narrative adaptation provides a variation of viewing experience with no loss of narrative comprehension.

The dominant interaction paradigm in Interactive Storytelling (IS) systems so far has been active interventions by the user by means of a variety of modalities. PINTER is an IS system that uses physiological inputs - surface electromyography (EMG) and galvanic skin response (GSR) [1] - as a form of passive interaction, opening up the possibility of the use of traditional filmic techniques [2, 3] to implement IS without requiring immersion-breaking interactive responses. The goal of this demonstration is to illustrate the ways in which passive interaction combined with filmic visualisation, dialogue and music, and a plan-based narrative generation approach can form a new basis for an adaptive interactive narrative.

Previous attempts at developing immersive versions of game engines have faced difficulties in achieving both overall high performance and preserving reusability of software developments. In this paper, we present a high-level VR middleware based on one of the most successful commercial game engines: the Unreal® Engine 3.0 (UE3). We describe a VR framework implemented as an extension to the Unreal® Development Kit (UDK) supporting CAVE"-like installations. Our approach relies on a distributed architecture reinforced by specific replication patterns to synchronize the user's point of view and interactions within a multi-screen installation. Our performance benchmarks indicated that our immersive port does not affect the game engine performance, even with complex real-time applications, such as fast-paced multiplayer First Person Shooter (FPS) games or high-resolution graphical environments with 2M+ polygons. A user study also demonstrated the capacity of our VR middleware to elicit high spatial presence while maintaining low cybersickness effects. With free distribution, we believe such a platform can support future Entertainment and VR research.

This paper describes an experiment comparinng immersive and non-immersive gaming using a state-of-the-art first person shooter game (FPS) in which we analyse user experience and performance through a combination of in-game metrics, questionaires and subjective reports. Our results show an overwhelming subjective preference for the immersive version despite a decrease in performance attributed to a more realistic aimming mechanism. Interaction metrics suggest that users took full advantage of the immmersive context rather than simply transposing their desktop gaming skills.

Virtual characters are at the epicentre of Interactive Storytelling systems and in recent years multiple AI planning approaches have been described to specify their autonomous behaviour. This demonstrator provides an overview of our novel approach to the definition of virtual characters aimed at achieving a balance between character autonomy and global plot structure which proposes the notion of a character's Point of View. Additionally, the demonstrator offers the active spectator the ability to discover the story described from the perspective of a number of different characters. We present our fully-implemented Interactive Narrative based on Shakespeare's Merchant of Venice. The system, which features a novel AI planning approach to story generation, can generate very different stories depending on the Point of View adopted and support dynamic modification of the story world which results in different story consequences.

In the paper we present a prototype of video-based storytelling that is able to generate multiple story variants from a baseline video. The video content for the system is generated by an adaptation of forefront video summarisation techniques that decompose the video into a number of Logical Story Units (LSU) representing sequences of contiguous and interconnected shots sharing a common semantic thread. Alternative storylines are generated using AI Planning techniques and these are used to direct the combination of elementary LSU for output. We report early results from experiments with the prototype in which the reordering of video shots on the basis of their high-level semantics produces trailers giving the illusion of different storylines.

We have seen ten years of the application of AI planning to the problem of narrative generation in Interactive Storytelling (IS). In that time planning has emerged as the dominant technology and has featured in a number of prototype systems. Nevertheless key issues remain, such as how best to control the shape of the narrative that is generated (e.g., by using narrative control knowledge , i.e., knowledge about narrative features that enhance user experience) and also how best to provide support for real-time interactive performance in order to scale up to more realistic sized systems. Recent progress in planning technology has opened up new avenues for IS and we have developed a novel approach to narrative generation that builds on this. Our approach is to specify narrative control knowledge for a given story world using state trajectory constraints and then to treat these state constraints as landmarks and to use them to decompose narrative generation in order to address scalability issues and the goal of real-time performance in larger story domains. This approach to narrative generation is fully implemented in an interactive narrative based on the “Merchant of Venice.” The contribution of the work lies both in our novel use of state constraints to specify narrative control knowledge for interactive storytelling and also our development of an approach to narrative generation that exploits such constraints. In the article we show how the use of state constraints can provide a unified perspective on important problems faced in IS.

In this paper, we describe a small-scale, yet complete, integration of a real-time immersive interactive storytelling system. While significant progress has been achieved in recent years on the individual component technologies of interactive storytelling, the main objective of this work is to investigate the concept of interactive storytelling in a fully immersive context. We describe each individual component of immersive interactive storytelling from a technical perspective. We have used a commercial game engine as a development environment, supporting real-time visualisation as well as the inclusion of Artificial Intelligence components controlling virtual actors. This visualisation engine has been ported to an immersive setting using dedicated software and hardware supporting real-time stereoscopic visualisation. The hardware platform is built around a 4-sided CAVE-like immersive display operated by a PC-cluster. The interactive storytelling engine is constituted by a planning system based on characters motivations and emotional states. The user can interact with the virtual world using multimodal interaction. We illustrate the system's behaviour on the implementation of excerpts from Madame Bovary, a classic XIXth century novel, and demonstrate the ability for the user to play the role of one of the characters and influence the unfolding of the story by his actions.

In this paper, we describe a mixed reality system based on a "magic mirror" model, in which the user's image is captured in real time by a video camera, extracted from his/her background and mixed with a 3D graphic model of a virtual image including the synthetic characters taking part in the story. The resulting image is projected on a large screen facing the user, who sees his/her own image embedded in the virtual stage with the synthetic actors. The graphic component of the mixed reality world is based on a game engine, Unreal Tournament 2003. This engine not only performs graphic rendering and character animation but incorporates a new version of our previously described storytelling engine. A single 2D camera facing the user analyses the image in real-time by segmenting the user's contours.

Abstract In this paper, we describe a method for implementing the behaviour of artificial actors in the context of interactive storytelling. We have developed a fully implemented prototype based on the Unreal Tournament™ game engine, and carried experiments with a simple sitcom‐like scenario. We discuss the central role of artificial actors in interactive storytelling and how real‐time generation of their behaviour participates in the creation of a dynamic storyline. We follow previous work describing the behaviour of artificial actors through AI planning formalisms, and adapt it to the context of narrative representation. In this context, the narrative equivalent of a character's behaviour consists of its role. The set of possible roles for a given actor is represented as a hierarchical task network (HTN). The system uses HTN planning to dynamically generate the character roles, by interleaving planning and execution, which supports dynamic interaction between actors, as well as user intervention in the unfolding plot. Finally, we present several examples of short plots and situations generated by the system from the dynamic interaction of artificial actors. Copyright © 2002 John Wiley & Sons, Ltd.

In this paper, we describe a method for implementing AI-based animation of artificial actors in the context of interactive storytelling. We have developed a fully implemented prototype based on the Unreal/sup TM/ game engine and carried out experiments with a simple sitcom-like scenario. We discuss the central role of artificial actors in interactive storytelling and how real-time generation of their behaviour participates to the creation of a dynamic storyline. We follow previous work describing the behaviour of artificial actors through AI planning formalisms, and adapt it to the context of narrative representation. The set of all possible behaviours, accounting for various instantiations of a basic plot, can be represented through an AND/OR graph. A real-time variant of the AO* algorithm can be used to interleave planning and action, thus allowing characters to interact between themselves and with the user. Finally, we present several examples of short plots and situations generated by the system from the dynamic interaction of artificial actors.