Recent RL research has utilized reward shaping--particularly complex shaping rewards such as intrinsic motivation (IM)--to encourage agent exploration in sparse-reward environments. While often effective, ``reward hacking'' can lead to the shaping reward being optimized at the expense of the extrinsic reward, resulting in a suboptimal policy. Potential-Based Reward Shaping (PBRS) techniques such as Generalized Reward Matching (GRM) and Policy-Invariant Explicit Shaping (PIES) have mitigated this. These methods allow for implementing IM without altering optimal policies. In this work we show that they are effectively unsuitable for complex, exploration-heavy environments with long-duration episodes. To remedy this, we introduce Action-Dependent Optimality Preserving Shaping (ADOPS), a method of converting intrinsic rewards to an optimality-preserving form that allows agents to utilize IM more effectively in the extremely sparse environment of Montezuma's Revenge. We also prove ADOPS accommodates reward shaping functions that cannot be written in a potential-based form: while PBRS-based methods require the cumulative discounted intrinsic return be independent of actions, ADOPS allows for intrinsic cumulative returns to be dependent on agents' actions while still preserving the optimal policy set. We show how action-dependence enables ADOPS's to preserve optimality while learning in complex, sparse-reward environments where other methods struggle.

Narrative planning is the use of automated planning to construct, communicate, and understand stories, a form of information to which human cognition and enaction is pre-disposed. We review the narrative planning problem in a manner suitable as an introduction to the area, survey different plan-based methodologies and affordances for reasoning about narrative, and discuss open challenges relevant to the broader AI community.

Extensive research has been conducted on using procedural music generation in real-time applications such as accompaniment to musicians, visual narratives, and games. However, less attention has been paid to the enhancement of textual narratives through music. In this paper, we present Mood Into Note Using Extracted Text (MINUET), a novel system that can procedurally generate music for textual narrative segments using sentiment analysis. Textual analysis of the flow and sentiment derived from the text is used as input to condition accompanying music. Music generation systems have addressed variations through changes in sentiment. By using an ensemble predictor model to classify sentences as belonging to particular emotions, MINUET generates text-accompanying music with the goal of enhancing a reader’s experience beyond the limits of the author’s words. Music is played via the JMusic library and a set of Markov chains specific to each emotion with mood classifications evaluated via stratified 10-fold cross validation. The development of MINUET affords the reflection and analysis of features that affect the quality of generated musical accompaniment for text. It also serves as a sandbox for further evaluating sentiment-based systems on both text and music generation sides in a coherent experience of an implemented and extendable experiential artifact.

In this work we test the hypothesis that interacting with an intelligent virtual character in Virtual Reality (VR) has a stronger impact compared to the same interaction in a traditional non-immersive platform, both in terms of presence and believability.

Joint attention refers to the shared focal points of attention for occupants in a space. In this work, we introduce a computational definition of joint attention for the automated editing of meetings in multi-camera environments from the AMI corpus. Using extracted head pose and individual headset amplitude as features, we developed three editing methods: (1) a naive audio-based method that selects the camera using only the headset input, (2) a rule-based edit that selects cameras at a fixed pacing using pose data, and (3) an editing algorithm using LSTM (Long-short term memory) learned joint-attention from both pose and audio data, trained on expert edits. The methods are evaluated qualitatively against the human edit, and quantitatively in a user study with 22 participants. Results indicate that LSTM-trained joint attention produces edits that are comparable to the expert edit, offering a wider range of camera views than audio, while being more generalizable as compared to rule-based methods.

The aim of conflict resolution education is to impart essential strategies and skills for resolving conflicts effectively. While these are important life skills, conflict resolution can be difficult to teach because it requires individuals to interact with others, explore new strategies, and receive feedback within a natural social context in order for strong connections to be made. As board games often involve co-located multiplayer interaction and can be effective tools for young learners, we explore the possibility of incorporating learning about conflict resolution into a tabletop game. We describe the process of designing an educational board game - StarStruck - that fosters discussions about conflict management via operationalization of conflict strategies drawn from an instrument founded in social psychology theory. Through in- and out-of-board interactions, StarStruck is designed to provide players with affordances to explore different resolution strategies within their natural social environment. We present examples from initial playtesting sessions to consider the expressive range of conflict scenarios generated by playing the game. This work serves as a preliminary illustration of how to map the vocabulary of conflict resolution to game mechanics, dynamics, and aesthetics so that players can naturally engage with and discuss conflict interactions.

Procedural narrative generation systems often focus on autonomous agent based simulations to create emergent interactions, plan-based approaches to provide guarantees for coherence, or using elements of simulation to guide plan-based approaches. These different approaches, with some exceptions, tend to trade off character autonomy in service of more designer controlled experiences or content authoring in service of encoding domain knowledge of possible branches of the narrative and participating characters. We have developed a system, called Stories of the Town, that automatically generates narratives by synthesizing three distinct approaches to traditional narrative generation: context-free grammars, planning, and simulation. More specifically, our system generates narratives via probabilistic context-free grammars applied to state-space planning problem solutions from planning problem formulations of simulated character models. Our system uses character simulations to generate variety in narratives and ensures narrative coherence through authoring probabilistic context-free grammars. By doing so, this system takes advantage of the strengths of each individual approach (e.g. controllability, scalability, intentionality, and variety) to generate narratives that are extensible, expressive, consistent with simulated character personalities and histories, and controllable. We show that this system has strong potential in automatically generating varied, complex, consistent, and goal-oriented narratives. Further development of the system will allow for more efficient utilization of the strengths of each narrative generation approach while also using these strengths to supplement their individual shortcomings.

An open challenge for AI in digital games is narrative experience management, the problem of automatically directing virtual characters in an interactive story to produce specific narrative effects for human participants. One important aspect of interactive narrative quality is participant choice, which provides the central distinction between interactive and linear storytelling. Most experience management work identifies storytelling patterns an agent should value or guides participants through high-value story trajectories. However, the types and quality of choices available in an interactive story are influenced not only by a model of narrative and specific story character actions, but also the world design that affords situations and actions. In this paper, we present a method for generating all unique, accessible choice combinations for a story world as a first step towards experience-driven interactive narrative world design. We benchmark the approach against several baselines and discuss its use as a tool for human and automated world designers.

While generative approaches to game design offer great promise, systems can only reliably generate what they can “understand,” which is often represented in a limited, implicit form in hand-crafted evaluation functions or constructive rules. Proceduralist readings, a semiformal approach for interpreting the meaning of a game based on its underlying processes and interactions in conjunction with aesthetic and cultural cues, offer a novel, systematic approach to game understanding. We formalize proceduralist argumentation as a logic program that performs static reasoning over game specifications to derive higher level meanings, as part of Gemini, a bidirectional game analysis and generation system.

Modern school environments are usually populated with children from diverse ethnic, cultural and social backgrounds, bringing in different social norms and skills, diverse behaviours and often contradicting cooperation strategies. As a result, conflicts are inevitable and should be resolved as quickly and painlessly as possible, making sure that school life and the learning process continue as intended. The Siren serious game aims to educate 10-14 year old students on conflict management and resolution, presenting them with user- and culture-adaptive mini game scenarios, based on popular game genres and taking into account their affective expressivity and in-game behaviour to adjust the intensity of the conflict to better suit their needs and competencies.

Goal-driven autonomy (GDA) is a conceptual model for creating an autonomous agent that monitors a set of expectations during plan execution, detects when discrepancies occur, builds explanations for the cause of failures, and formulates new goals to pursue when planning failures arise. While this framework enables the development of agents that can operate in complex and dynamic environments, implementing the logic for each of the subtasks in the model requires substantial domain engineering. We present a method using case-based reasoning and intent recognition in order to build GDA agents that learn from demonstrations. Our approach reduces the amount of domain engineering necessary to implement GDA agents and learns expectations, explanations, and goals from expert demonstrations. We have applied this approach to build an agent for the real-time strategy game StarCraft. Our results show that integrating the GDA conceptual model into the agent greatly improves its win rate.

Confronting conflicts and coping with them is part of social life, since conflicts seem to arise in almost every context and developmental stage of human life. The personal and collective gains that follow conflict resolution have motivated scholars across many research fields to advocate the use of pro-social mechanisms for resolution. The Siren serious game aims to support teachers' role to educate young people on how to resolve conflicts, by employing affect-aware, user- and cultural adaptivity to provide interesting and relevant conflict scenarios and resolution approaches.

Video games are increasingly producing huge datasets available for analysis resulting from players engaging in interactive environments. These datasets enable investigation of individual player behavior at a massive scale, which can lead to reduced production costs and improved player retention. We present an approach for modeling player retention in Madden NFL 11, a commercial football game. Our approach encodes gameplay patterns of specific players as feature vectors and models player retention as a regression problem. By building an accurate model of player retention, we are able to identify which gameplay elements are most influential in maintaining active players. The outcome of our tool is recommendations which will be used to influence the design of future titles in the Madden NFL series.

In this paper, we present the design, implementation, and evaluation of an end-to-end camera planning system called Darshak. Darshak automatically constructs cinematic narrative discourse of a given story in a 3-D virtual environment. It utilizes a hierarchical partial-order causal link (POCL) planning algorithm to generate narrative plans that contain story events and camera directives for filming them. Dramatic situation patterns, commonly used by writers of fictional narratives, are formalized as communicative plan operators that provide a basis for structuring the cinematic content of the story's visualization. The dramatic patterns are realized through abstract communicative operators that represent operations on a viewer's beliefs about the story and its telling. Camera shot compositions and transitions are defined in this plan-based framework as execution primitives. Darshak's performance is evaluated through a novel user study based on techniques used to evaluate existing cognitive models of narrative comprehension. Initial study reveals significant effect of the choice of visualization strategies on measured viewer comprehension. It further shows significant effect of Darshak's choice of visualization strategy on comprehension.

We propose a novel evaluation methodology for intelligent camera control systems based on established techniques of measuring story comprehension from cognitive psychology. The proposed methodology can be used specifically for evaluating the effectiveness of the camera system in communicating the story. We introduce a psychological model of question answering called QUEST and present a preliminary evaluation design of videos automatically generated by Darshak, an intelligent cinematic camera planning system. Initial results from our analysis are encouraging and motivate further work in evaluation of intelligent camera control systems as well as cognitive models of story comprehension through the visual medium.