Distributed Deep Reinforcement Learning (DRL) aims to leverage more computational resources to train autonomous agents with less training time. Despite recent progress in the field, reproducibility issues have not been sufficiently explored. This paper first shows that the typical actor-learner framework can have reproducibility issues even if hyperparameters are controlled. We then introduce Cleanba, a new open-source platform for distributed DRL that proposes a highly reproducible architecture. Cleanba implements highly optimized distributed variants of PPO and IMPALA. Our Atari experiments show that these variants can obtain equivalent or higher scores than strong IMPALA baselines in moolib and torchbeast and PPO baseline in CleanRL. However, Cleanba variants present 1) shorter training time and 2) more reproducible learning curves in different hardware settings.

Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks.

Monte Carlo Tree Search (MCTS) employs Multi-Armed Bandit (MAB) techniques to direct the policy for child node selection during tree construction. Typical MCTS implementations have relied on the Upper Confidence Bounds for Trees (UCT) strategy, which leverages a specific variant of the general Upper Confidence Bounds (UCB) approach. The success of such strategies relies heavily on the proper tuning of the UCB C parameter to guide exploration effectively. This paper examines (1) the advantages of per-arm tuning of C, (2) the potential for a parameter-less UCB variant called UCBT to provide opportunities for automatic derivation of effective C values without prior tuning in a strategy called Poly-UCB1, and (3) the application of both of these concepts toward operational tuning of C during MCTS node expansion and tree construction in a strategy called UCB-Multi. We evaluate our approach in three turn-based, adversarial board games.

Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Algorithmic fairness is an essential requirement as AI becomes integrated in society. In the case of social applications where AI distributes resources, algorithms often must make decisions that will benefit a subset of users, sometimes repeatedly or exclusively, while attempting to maximize specific outcomes. How should we design such systems to serve users more fairly? This paper explores this question in the case where a group of users works toward a shared goal in a social exergame called Step Heroes. We identify adverse outcomes in traditional multi-armed bandits (MABs) and formalize the Greedy Bandit Problem. We then propose a solution based on a new type of fairness-aware multi-armed bandit, Shapley Bandits. It uses the Shapley Value for increasing overall player participation and intervention adherence rather than the maximization of total group output, which is traditionally achieved by favoring only high-performing participants. We evaluate our approach via a user study (n=46). Our results indicate that our Shapley Bandits effectively mediates the Greedy Bandit Problem and achieves better user retention and motivation across the participants.

Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing.In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization.We identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in many compositional tasks.We achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

This article focuses ontracing player knowledgein educational games. Specifically, given a set of concepts or skills required to master a game, the goal is to estimate the likelihood with which the current player has mastery of each of those concepts or skills. The main contribution of the work is an approach that integrates machine learning and domain knowledge rules to find when the player applied a certain skill and either succeeded or failed. This is then given as input to a standard knowledge tracing module (such as those from intelligent tutoring systems) to perform knowledge tracing. We evaluate our approach in the context of an educational game calledParallelto teach parallel and concurrent programming with data collected from real users, showing our approach can predict students skills with a low mean-squared error. We also provide results from deployment of our system in a classroom environment.

Real-time strategy games are challenging problems from an AI point of view. Specifically, they are particularly hard for tree search algorithms due to their combinatorial branching factors and the limited amount of time available to choose actions. As the community grows and many real-time strategy (RTS) game competitions are being held, much work has been done in the direction of integrating hand-authored scripted bots into tree search algorithms, with the goal of making search tractable. In this article, we survey a collection of representative algorithms which integrates scripts into search or planning algorithms. Then we compare them experimentally in the$\mu$RTS environment and examine the tradeoffs for designing such algorithms. We also discuss the potential future work in this direction of research and connections to other types of algorithms.

This paper focuses on player modeling in multiplayer adaptive games. While player modeling has received a significant amount of attention, less is known about how to use player modeling in multiplayer games, especially when an experience management AI must make decisions on how to adapt the experience for the group as a whole. Specifically, we present a multi-armed bandit (MAB) approach for modeling groups of multiple players. Our main contributions are a new MAB framework for multiplayer modeling and techniques for addressing the new challenges introduced by the multiplayer context, extending previous work on MAB-based player modeling to account for new group-generated phenomena not present in single-user models. We evaluate our approach via simulation of virtual players in the context of multiplayer adaptive exergames.

In recent years, researchers have achieved great success in applying Deep Reinforcement Learning (DRL) algorithms to Real-time Strategy (RTS) games, creating strong autonomous agents that could defeat professional players in StarCraft II. However, existing approaches to tackle full games have high computational costs, usually requiring the use of thousands of GPUs and CPUs for weeks. This paper has two main contributions to address this issue: 1) We introduce Gym-JLRTS (pronounced “gym-micro-RTS”) as a fast-to-run RL environment for full-game RTS research and 2) we present a collection of techniques to scale DRL to play full-game µRTS as well as ablation studies to demonstrate their empirical importance. Our best-trained bot can defeat every µRTS bot we tested from the past µRTS competitions when working in a single-map setting, resulting in a state-of-the-art DRL agent while only taking about 60 hours of training using a single machine (one GPU, three vCPU. 16GB RAM).

The contextual bandit problem is a richer framework than stochastic bandits that has many applications since it allows the learner has access to additional information (the “context”). This additional information can help predict the expected utility of the different arms in many cases. Moreover, combinatorial bandits are a class of bandit problem where the space of possible arms to choose from has a combinatorial structure. In this paper, we investigate the bandit problem where we have both contextual information and there is a combinatorial arm structure, which we call contextual combinatorial bandits (CCMABs). We apply contextual combinatorial bandits to realtime strategy (RTS) games, and study different algorithms to solve CCMABs with different trade-offs of computational efficiency and learning biases. Specifically, we focus on the problem of determining map-specific game playing policies, and formulate it as a CCMABs.

As more and more online search queries come from voice, automatic speech recognition becomes a key component to deliver relevant search results. Errors introduced by automatic speech recognition (ASR) lead to irrelevant search results returned to the user, thus causing user dissatisfaction. In this paper, we introduce an approach, "Mondegreen", to correct voice queries in text space without depending on audio signals, which may not always be available due to system constraints or privacy or bandwidth (for example, some ASR systems run on-device) considerations. We focus on voice queries transcribed via several proprietary commercial ASR systems. These queries come from users making internet, or online service search queries. We first present an analysis showing how different the language distribution coming from user voice queries is from that in traditional text corpora used to train off-the-shelf ASR systems. We then demonstrate that Mondegreen can achieve significant improvements in increased user interaction by correcting user voice queries in one of the largest search systems in Google. Finally, we see Mondegreen as complementing existing highly-optimized production ASR systems, which may not be frequently retrained and thus lag behind due to vocabulary drifts.

Social comparison-based features are widely used in social computing apps. However, most existing apps are not grounded in social comparison theories and do not consider individual differences in social comparison preferences and reactions. This paper is among the first to automatically personalize social comparison targets. In the context of an m-health app for physical activity, we use artificial intelligence (AI) techniques of multi-armed bandits. Results from our user study (n=53) indicate that there is some evidence that motivation can be increased using the AI-based personalization of social comparison. The detected effects achieved small-to-moderate effect sizes, illustrating the real-world implications of the intervention for enhancing motivation and physical activity. In addition to design implications for social comparison features in social apps, this paper identified the personalization paradox, the conflict between user modeling and adaptation, as a key design challenge of personalized applications for behavior change. Additionally, we propose research directions to mitigate this Personalization Paradox.

This paper explores multi-armed bandit (MAB) strategies in very short horizon scenarios, i.e., when the bandit strategy is only allowed very few interactions with the environment. This is an understudied setting in the MAB literature with many applications in the context of games, such as player modeling. Specifically, we pursue three different ideas. First, we explore the use of regression oracles, which replace the simple average used in strategies such as ε-greedy with linear regression models. Second, we examine different exploration patterns such as forced exploration phases. Finally, we introduce a new variant of the UCB1 strategy called UCBT that has interesting properties and no tunable parameters. We present experimental results in a domain motivated by exergames, where the goal is to maximize a player's daily steps. Our results show that the combination of ε-greedy or ε-decreasing with regression oracles outperforms all other tested strategies in the short horizon setting.

Real-Time Strategy (RTS) games are an interesting environment to study challenging AI problems, such as real-time adversarial planning and opponent modeling. In this paper we focus on approaches that make use of replay data, which usually encode domain expert knowledge of gameplay. Some of these approaches use supervised learning to learn player/agent strategy models and thus rely on these replays being annotated with specific strategies or other labels. However, replays do not usually contain labels for these strategies. The problem we address in this paper is the automatic discovery of meaningful labeling of replays in RTS games. We address this problem by learning action and replay embeddings via recursive neural network models such as LSTMs. These embedded replays can then be clustered to discover labelings by using the clusters as the labels. We show that we can learn embeddings and discover labelings for replays that are correlated with meaningful information from those replays.

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

This paper focuses on building personalized player models solely from player behavior in the context of adaptive games. We present two main contributions: The first is a novel approach to player modeling based on multi-armed bandits (MABs). This approach addresses, at the same time and in a principled way, both the problem of collecting data to model the characteristics of interest for the current player and the problem of adapting the interactive experience based on this model. Second, we present an approach to evaluating and fine-tuning these algorithms prior to generating data in a user study. This is an important problem, because conducting user studies is an expensive and labor-intensive process; therefore, an ability to evaluate the algorithms beforehand can save a significant amount of resources. We evaluate our approach in the context of modeling players’ social comparison orientation (SCO) and present empirical results from both simulations and real players.

Computer games represent an ideal research domain for the next generation of personalized digital applications. This paper presents a player-centered framework of AI for game personalization, complementary to the commonly used system-centered approaches. Built on the Structure of Actions theory, the paper maps out the current landscape of game personalization research and identifies eight open problems that need further investigation. These problems require deep collaboration between technological advancement and player experience design.

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

This paper focuses on the problem of scaling Combinatory Categorial Grammar (CCG)-based plan recognition to large CCG representations in the context of Real-Time Strategy (RTS) games. Specifically, we present a technique to scale plan recognition to large domain representations using Monte-Carlo Tree Search (MCTS). CCG-based planning and plan recognition (like other domain-configurable planning frameworks) require domain definitions to be either manually authored or learned from data. Prior work has demonstrated successful learning of these CCG domain definitions from data, but these representations can be very large for complex application domains. We propose a MCTS-based approach to search for explanations and predict the goal of a given sequence of observed actions. We evaluate our approach on the RTS game AI testbed microRTS. Our experimental results show our method scales better to these large, learned CCGs than previous CCG-based approaches.

Experience Management studies AI systems that automatically adapt interactive experiences such as games to tailor to specific players and to fulfill design goals. Although it has been explored for several decades, existing work in experience management has mostly focused on single-player experiences. This paper is a first attempt at identifying the main challenges to expand EM to multi-player/multi-user games or experiences. We also make connections to related areas where solutions for similar problems have been proposed (especially group recommender systems) and discusses the potential impact and applications of multi-player EM.

Concurrent and parallel programming (CPP) skills are increasingly important in today's world of parallel hardware. However, the conceptual leap from deterministic sequential programming to CPP is notoriously challenging to make. Our educational game Parallel is designed to support the learning of CPP core concepts through a game-based learning approach, focusing on the connection between gameplay and CPP. Through a 10-week user study (n 25) in an undergraduate concurrent programming course, the first empirical study for a CPP educational game, our results show that Parallel offers both CPP knowledge and student engagement. Furthermore, we provide a new framework to describe the design space for programming games in general.

Voice User Interfaces (VUIs) are becoming increasingly popular. However, how VUIs can adapt to user differences remains insufficiently understood. We analyze usage data from a user study (n=50) where participants interacted with an unfamiliar VUI. Through automated clustering and statistical analysis, we present user models of their behavior patterns. We found user behavior can be grouped into three clusters: people who become proficient with the system and typically stay proficient while completing different tasks, people who exhibit an exploratory approach to completing tasks, and people who struggled to complete tasks. We discuss design implications based on these behavior clusters.

In parallel programming, there is a shift away from the single execution path of sequential programming to situations where non-deterministic operation force consideration of multiple paths of execution. Compared to the substantial computer science education literature on helping students to learn sequential programming, there are fewer studies of the cognitive difficulties that students follow when learning parallel programming. To address this, we created a computer game, Parallel involving concurrent situations. The game is an abstract representation of concurrency problems where players are asked to solve a progression of puzzles involving arrows moving concurrently on tracks. Play does not require coding. The goals of our research were to 1) explore how students acquire skills in the design of solutions with parallelism, and 2) explore how interactive games can substitute or compliment conventional parallel programming courses. Through two user studies of the game (n=7) where students played the game and used a talk-aloud protocol alongside a researcher, three major themes emerged, that of non-determinism where students were able to make the connection of non-deterministic behavior in parallel programming to the game, self-efficacy where students were stating they felt their knowledge of parallel programming increased after playing the game, and expertise where researchers learned that expertise was important to successful connection of the game to parallel programming concepts These findings show that students are beginning to see the connection between the game/s presentation of concurrency to programming concepts such as non-determinism.

Game-tree search algorithms, such as Monte Carlo tree search, require access to a forward model (or “simulator”) of the game at hand. However, in some games such forward model is not readily available. This paper presents three forward models for two-player attrition games, which we call “combat models,” and show how they can be used to simulate combat in real-time strategy games. We also show how these combat models can be learned from replay data. We use STARCRAFT as our application domain. We report experiments comparing our combat models predicting combat results and their impact when used for tactical decisions during a real game.

Evaluating the output of content generators is still one of the key open research challenges in Procedural Content Generation (PCG). This paper presents a collection of metrics for evaluating the quality of platform game levels, and analyzes how well these metrics are able to capture the human-perceived difficulty, visual aesthetics and enjoyment of these levels. We show empirically, in the context of Infinite Mario Bros (IMB), that some of the proposed metrics yield correlation values with human ratings that are near empirical upper bounds derived from a human inter-rater agreement study. We also show that a simple linear regression model using a subset of our metrics as input features is able to substantially outperform a previous approach that uses a neural network for predicting human-perceived difficulty, visual aesthetics, and enjoyment in IMB levels.

In the context of a learning game to teach parallel programming, we describe a procedural content generation (PCG) approach that can be controlled to generate programming puzzles involving a desired set of concepts, and of desired size and "difficulty". Our approach is based on grammars to control the generation of the puzzle structure, and orthographic graph embedding techniques to render it into a two-dimensional grid for our game. The proposed PCG system is designed to work with a player model in order to provide personalized learning experiences. We present an evaluation of the variability of the generated puzzles using several metrics including challenge and solvability as evaluated by a custom-build model checker. Our evaluation shows that this PCG system can generate a large number of varied puzzles but it is still not able to generate puzzles with certain aesthetic and functional qualities found in puzzles generated by human authors.

In this paper we present a survey of two of the main areas of research within the field of computational narrative, namely narrative analysis and generation. We argue that there is a gap between these two lines of work and propose a taxonomy of the computational models of narrative used within each. We outline potential mappings between computational narrative models with the goal of bridging this gap and alleviating the authorial bottleneck problem occurring when authoring content for computational narrative generation systems. Finally we discuss related work in this direction and report on our work-in-progress towards an end-to-end computational narrative system to bridge the gap.

The use of statistical and machine learning approaches, such as Markov chains, for procedural content generation (PCG) has been growing in recent years in the field of Game AI. However, there has been little work in learning to generate content, specifically levels, accounting for player movement within those levels. We are interested in extracting player models automatically from play traces and using those learned models, paired with a machine learning-based generator to create levels that allow the same types of movements observed in the play traces. We test our approach by generating levels for Super Mario Bros. We compare our results against the original levels, a previous constrained sampling approach, and a previous approach that learned a combined player and level model.

Abstract Plan similarity measures play a key role in many areas of artificial intelligence, such as case‐based planning, plan recognition, ambient intelligence, or digital storytelling. In this paper, we present 2 novel structural similarity measures to compare plans based on a search process in the space of partial plans. Partial plans are compact representations of sets of plans with some common structure and can be organized in a lattice so that the most general partial plans are above the most specific ones. To compute our similarity measures, we traverse this space of partial plans from the most general to the most specific using successive refinements. Our first similarity measure is designed for propositional plan formalisms, and the second is designed for classical planning formalisms (including variables and types). We also introduce 2 novel refinement operators used to traverse the space of plans: an ideal downward refinement operator for propositional partial plans and a finite and complete downward refinement operator for classical partial plans. Finally, we evaluate our similarity measures in the context of a nearest neighbor classifier using 2 datasets commonly used in the plan recognition literature (Linux and Monroe), showing good results in both synthetic and real data.

Games with large branching factors pose a significant challenge for game tree search algorithms. In this paper, we address this problem with a sampling strategy for Monte Carlo Tree Search (MCTS) algorithms called "naive sampling", based on a variant of the Multi-armed Bandit problem called "Combinatorial Multi-armed Bandits" (CMAB). We analyze the theoretical properties of several variants of naive sampling, and empirically compare it against the other existing strategies in the literature for CMABs. We then evaluate these strategies in the context of real-time strategy (RTS) games, a genre of computer games characterized by their very large branching factors. Our results show that as the branching factor grows, naive sampling outperforms the other sampling strategies.

Procedural content generation has become a popular research topic in recent years. However, most content generation systems are specialized to a single game. We are interested in methods that can generate content for a wide variety of games without a game-specific algorithm design. Statistical approaches are a promising avenue for such generators and, more specifically, map generators. In this paper, we explore Markov models as a means of modeling and generating content for multiple domains. We apply our Markov models to Super Mario Bros., Loderunner , and Kid Icarus in order to determine how well our models perform in terms of the playability of the content generated, the expressive ranges of the models, and the effects of training data on those expressive ranges.

In this paper, we present our method for automatically extracting narrative information of characters and their narrative roles from natural language stories. In our corpus of 15 unannotated folk tales, our Voz system identifies 87% of the characters in the stories and correctly assigns 68% of the character roles. To better understand the sources of error in our system, we present an analytical methodology to study how the error is introduced by different modules and how it propagates through the pipeline. This methodology allows us to identify the bottleneck with the largest impact on the final error, which might be different from the module with the largest individual error in isolation. Our methodology can be applied to a wide variety of similar information extraction pipelines.

Labeling problems are finding increasing applications to optimization problems. They usually get realized into linear or quadratic optimization problems, which are inefficient for large graphs. In this paper we propose an efficient primal-dual solution, MLPD, for a family of labeling problems. We apply this algorithm to the analysis of immune repertoires, and compare it against our baseline approach based on refinement operators. We provide a comparative evaluation both in terms of accuracy and computational efficiency with respect to the baseline model, as well as to quadratic optimization.

Despite their use in traditional storytelling, analogy-based narrative devices have not been sufficiently explored in computational narrative. In this paper, we present our analogy-based story generation (ASG) approach in the Riu system, focusing on analogical retrieval and projection. We report on an empirical user evaluation about Riu's capability to retrieve and generate short noninteractive stories using the story analogies through mapping (SAM) algorithm. This work provides the foundation for exploration of ASG in more complex and interactive computational narrative works.

This paper presents an overview of the existing work on AI for real-time strategy (RTS) games. Specifically, we focus on the work around the game StarCraft, which has emerged in the past few years as the unified test bed for this research. We describe the specific AI challenges posed by RTS games, and overview the solutions that have been explored to address them. Additionally, we also present a summary of the results of the recent StarCraft AI competitions, describing the architectures used by the participants. Finally, we conclude with a discussion emphasizing which problems in the context of RTS game AI have been solved, and which remain open.

We present a novel ensemble architecture for learning problem-solving techniques from a very small number of expert solutions and demonstrate its effectiveness in a complex real-world domain. The key feature of our “Generalized Integrated Learning Architecture” (GILA) is a set of heterogeneous independent learning and reasoning (ILR) components, coordinated by a central meta-reasoning executive (MRE). The ILRs are weakly coupled in the sense that all coordination during learning and performance happens through the MRE. Each ILR learns independently from a small number of expert demonstrations of a complex task. During performance, each ILR proposes partial solutions to subproblems posed by the MRE, which are then selected from and pieced together by the MRE to produce a complete solution. The heterogeneity of the learner-reasoners allows both learning and problem solving to be more effective because their abilities and biases are complementary and synergistic. We describe the application of this novel learning and problem solving architecture to the domain of airspace management, where multiple requests for the use of airspaces need to be deconflicted, reconciled, and managed automatically. Formal evaluations show that our system performs as well as or better than humans after learning from the same training data. Furthermore, GILA outperforms any individual ILR run in isolation, thus demonstrating the power of the ensemble architecture for learning and problem solving.

In computer-based interactive narrative, a key challenge is the conflict between user agency and authorial control of the story quality. In this paper, we use the constructs of character status and status shifts from improvisational and interactive theatre to further engage users in the creative process of co-creating the story. Based on the cognitive semantics theory of force dynamics, we develop a computational model of status shifts.

When compared to the depiction of external actions, modern computer games have developed very limited means of conveying game characters' inner activities. In this paper, we focus on different narrative perspectives and the ways in which they enable us to express a wider range of characters' inner worlds. We also present our on-going project Remembrance which uses a system of shifting external environments to reflect the character's inner world.

Computational narrative is a complex and interesting domain for exploring AI techniques that algorithmically analyze, understand, and most importantly, generate stories. This paper studies the importance of domain knowledge in story generation, and particularly in analogy-based story generation (ASG). Based on the construct of knowledge container in case-based reasoning, we present a theoretical framework for incorporating domain knowledge in ASG. We complement the framework with empirical results in our existing system Riu.

In this paper we propose an argumentation-based framework for multiagent induction, where two agents learn separately from individual training sets, and then engage in an argumentation process in order to converge to a common hypothesis about the data. The result is a multiagent induction strategy in which the agents minimize the set of cases that they have to exchange (using argumentation) in order to converge to a shared hypothesis. The proposed strategy works for any induction algorithm which expresses the hypothesis as a collection of rules. We show that the strategy converges to a hypothesis indistinguishable in training set accuracy from that learned by a centralized strategy.

Some domains, such as real‐time strategy (RTS) games, pose several challenges to traditional planning and machine learning techniques. In this article, we present a novel on‐line case‐based planning architecture that addresses some of these problems. Our architecture addresses issues of plan acquisition, on‐line plan execution, interleaved planning and execution, and on‐line plan adaptation. We also introduce the Darmok system, which implements this architecture to play Wargus (an open source clone of the well‐known RTS game Warcraft II). We present empirical evaluation of the performance of Darmok and show that it successfully learns to play the Wargus game.

A growing research community is working toward employing drama management components in story‐based games. These components gently guide the story toward a narrative arc that improves the player's gaming experience. In this article we evaluate a novel drama management approach deployed in an interactive fiction game called Anchorhead. This approach uses player's feedback as the basis for guiding the personalization of the interaction. The results indicate that adding our Case‐based Drama manaGer (C‐DraGer) to the game guides the players through the interaction and provides a better overall player experience. Unlike previous approaches to drama management, this article focuses on exhibiting the success of our approach by evaluating results using human players in a real game implementation. Based on this work, we report several insights on drama management which were possible only due to an evaluation with real players.

Learning agents can improve performance cooperating with other agents, particularly learning agents forming a committee outperform individual agents. This "ensemble effect" is well known for multi-classifier systems in Machine Learning. However, multi-classifier systems assume all data is known to all classifiers while we focus on agents that learn from cases (examples) that are owned and stored individually. In this article we focus on how individual agents can engage in bargaining activities that improve the performance of both individual agents and the committee. The agents are capable of self-evaluation and determining that some data used for learning is unnecessary. This "refuse" data can then be exploited by other agents that might found some part of it profitable to improve their performance. The experiments we performed show that this approach improves both individual and committee performance and we analyze how these results in terms of the "ensemble effect".