Multi-agent reinforcement learning has shown promise in learning cooperative behaviors in team-based environments. However, such methods often demand extensive training time, which inhibits their application for game-AI in standard game development. For instance, the state-of-the-art method TiZero takes 40 days to train high-quality policies for a football environment. In this paper, we hypothesize that better exploration mechanisms can improve the sample efficiency of multi-agent methods. Thereby, we propose utilizing a random network distillation bonus within the multi-agent TiZero framework, aiming to promote exploration. Additionally, we introduce architectural modifications to the original algorithm to enhance TiZero’s computational efficiency. We evaluate the sample efficiency of our approach against original TiZero through extensive experiments. Our results show that random network distillation improves the sample efficiency per training phase by 13.3% compared with the original TiZero, enhancing generalization and adaptability to previously difficult scenarios. This highlights the better applicability of our variant in practical game development settings. Lastly, we qualitatively evaluate the gameplay of the produced models against a heuristic AI. We find that random network distillation leads to a higher accuracy in shooting, and it achieves higher behavioral stability as shown by the lower standard deviation achieved in gameplay metrics. The code is available at https://github.com/electronicarts/marling.

In strategy games, one of the most important aspects of game design is maintaining a sense of challenge for players. Many mobile titles feature quick gameplay loops that allow players to progress steadily, requiring an abundance of levels and puzzles to prevent them from reaching the end too quickly. As with any content creation, testing and validation are essential to ensure engaging gameplay mechanics, enjoyable game assets, and playable levels. In this paper, we propose an automated approach that can be leveraged for gameplay testing and validation that combines traditional scripted methods with reinforcement learning, reaping the benefits of both approaches while adapting to new situations similarly to how a human player would. We test our solution on a popular tower defense game, Plants vs. Zombies. The results show that combining a learned approach, such as reinforcement learning, with a scripted AI produces a higher-performing and more robust agent than using only heuristic AI, achieving a $57.12 \%$ success rate compared to 47.95% in a set of 40 levels. Moreover, the results demonstrate the difficulty of training a general agent for this type of puzzle-like game.

Generative models for level generation have shown great potential in game production. However, they often provide limited control over the generation, and the validity of the generated levels is unreliable. Despite this fact, only a few approaches that learn from existing data provide the users with ways of controlling the generation, simultaneously addressing the generation of unsolvable levels. This article proposes autovalidated level generation, a novel method to improve models that learn from existing level designs using difficulty statistics extracted from gameplay. In particular, we use a conditional variational autoencoder to generate layouts for match-3 levels, conditioning the model on precollected statistics, such as game mechanics like difficulty, and relevant visual features, such as size and symmetry. Our method is general enough that multiple approaches could potentially be used to generate these statistics. We quantitatively evaluate our approach by comparing it to an ablated model without difficulty conditioning. In addition, we analyze both quantitatively and qualitatively whether the style of the dataset is preserved in the generated levels. Our approach generates more valid levels than the same method without difficulty conditioning.

This article proposes a novel deep reinforcement learning algorithm to perform automated analysis and detection of gameplay issues in complex 3-D navigation environments. The curiosity-conditioned proximal trajectories (CCPT) method combines curiosity and imitation learning to train agents that methodically explore in the proximity of known trajectories derived from expert demonstrations. We show how our new algorithm can explore complex environments, discovering gameplay issues, and design oversights in the process, and recognize and highlight them directly to game designers. We also propose a visual analytics interface to aid interpretation of results from the method. This interface transforms information from complex models into interpretable and interactive visual forms. We further demonstrate the effectiveness of the algorithm in a novel 3-D navigation environment, which reflects the complexity of modern video games. Our results show a higher level of coverage and bug discovery than baseline methods, demonstrating that our method can be a useful tool for game designers to automatically identify design issues. Moreover, our experiments show that the visual explanations provided by the analytics interface result in a significant increase in user trust and acceptance of automated playtesting and increased confidence in the use of machine learning techniques for video game development.

Going from research to production, especially for large and complex software systems, is fundamentally a hard problem. In large-scale game production, one of the main reasons is that the development environment can be very different from the final product. In this technical paper we describe an effort to add an experimental reinforcement learning system to an existing automated game testing solution based on scripted bots in order to increase its capacity. We report on how this reinforcement learning system was integrated with the aim to increase test coverage similar to [1] in a set of AAA games including Battlefield 2042 and Dead Space (2023). The aim of this technical paper is to show a use-case of leveraging reinforcement learning in game production and cover some of the largest time sinks anyone who wants to make the same journey for their game may encounter. Furthermore, to help the game industry to adopt this technology faster, we propose a few research directions that we believe will be valuable and necessary for making machine learning, and especially reinforcement learning, an effective tool in game production.

In games, as in many other domains, design validation and testing is a significant challenge as systems are growing in size and manual testing is becoming infeasible. In this position paper we outline an approach to automated game validation based on an imitation learning technique, and provide an analysis of the potential benefits to automated game testing. The method leverages a data-driven technique, which requires little effort and time and no knowledge of machine learning or programming, that designers can use to efficiently train game testing agents. We evaluate the validity of our claim by conducting a user study with industry experts. The survey results presented in this paper demonstrate the potential of a data-driven approach to reduce effort and enhance the quality of game testing. Moreover, the survey reveals several open challenges. To this end, we analyze the identified challenges and provide a basis for further research and discussion, as well as to help guide the development of imitation learning for game testing.

Training RL agents to solve novel environments is a notoriously difficult task. Here we present a new approach ARLPCG: Adversarial Reinforcement Learning for Procedural Content Generation, which procedurally generates and tests previously unseen environments with an auxiliary input as a control variable. The procedurally generated environments induces state diversity which increases the generalizability of the trained agents. ARLPCG deploys an adversarial model with one PCG RL agent (called Generator) and one solving RL agent (called Solver). The Generator receives a reward signal based on the Solver's performance, which encourages the environment design to be challenging but not impossible. To further drive diversity and control of the environment generation, we propose using auxiliary inputs for the Generator. The benefit is two-fold: Firstly, the Solver achieves better generalization through the Generator's generated challenges. Secondly, the trained Generator can be used as a creator of novel environments that, together with the Solver, can be shown to be solvable. We create two types of 3D environments to validate our model, representing two popular game genres: a third-person platformer and a racing game. In these cases, we show that ARLPCG has a significantly better solve ratio, and that the auxiliary inputs renders the levels creation controllable to a certain degree. For a video compilation of the results please visit https://youtu.be/z7q2PtVsT0I.

As modern games continue growing both in size and complexity, it has become more challenging to ensure that all the relevant content is tested and that any potential issues are properly identified and fixed. Attempting to maximize testing coverage using only human participants, however, results in a tedious and hard to orchestrate process which normally slows down the development cycle. Complementing playtesting via autonomous agents has shown great promise accelerating and simplifying this process. This paper addresses the problem of automatically exploring and testing a given scenario using reinforcement learning agents trained to maximize game state coverage. Each of these agents is rewarded based on the novelty of its actions, thus encouraging a curious and exploratory behaviour on a complex 3D scenario where previously proposed exploration techniques perform poorly. The curious agents are able to learn the complex navigation mechanics required to reach the different areas around the map, thus providing the necessary data to identify potential issues. Moreover, the paper also investigates different visualization strategies and evaluates how to make better use of the collected data to drive design decisions and to recognize possible problems and oversights.

General game testing relies on the use of human play testers, play test scripting, and prior knowledge of areas of interest to produce relevant test data. Using deep reinforcement learning (DRL), we introduce a self-learning mechanism to the game testing framework. With DRL, the framework is capable of exploring and/or exploiting the game mechanics based on a user-defined, reinforcing reward signal. As a result, test coverage is increased and unintended game play mechanics, exploits and bugs are discovered in a multitude of game types. In this paper, we show that DRL can be used to increase test coverage, find exploits, test map difficulty, and to detect common problems that arise in the testing of first-person shooter (FPS) games.