Quality Diversity (QD) is a branch of evolutionary computation that seeks high-quality and behaviorally diverse solutions to a problem. While adversarial problems are common, classical QD cannot be easily applied to them, as both fitness and behavior depend on the opposing solutions. Recently, Generational Adversarial MAP-Elites (GAME) has been proposed to coevolve both sides of an adversarial problem by alternating the execution of a multi-task QD algorithm against previous elites, called tasks. The original algorithm selects new tasks based on a behavioral criterion, which may lead to undesired dynamics due to inter-side dependencies. In addition, comparing sets of solutions cannot be done directly using classical QD metrics due to inter-side dependencies. In this paper, we propose (1) 6 metrics of adversarial quality and diversity based on an inter-variants tournament to compare the sets of solutions, ensuring a fair comparison, and (2) propose two tournament-informed task selection methods to promote higher quality and diversity at each generation. We evaluate the variants across three adversarial problems: Pong, a Cat-and-mouse game, and a Pursuers-and-evaders game. We show that the tournament-informed task selection method leads to higher adversarial quality and diversity. We hope that this work will help further advance adversarial QD.

We are in the midst of large-scale industrial and academic efforts to automate the processes of scientific, technological and creative production through AI-driven assistants. Historically, a fundamental property of these processes in their human form has been their open-endedness: their capacity for generating a seemingly endless supply of novel and meaningful new forms. Do artificial agents have any capacity for such fruitful unguided discovery? To answer this question, we turn to Picbreeder, the canonical exemplar of human-driven open-ended search, in which users collaboratively generated a diverse library of images through interactive evolution of small neural networks. We replicate Picbreeder, replacing human users with frontier Vision Language Models (VLMs). We observe clear qualitative differences between the output of our system and the historical human baseline, and attempt to characterize them using metrics of phylogenetic complexity and visual and semantic salience and novelty. In an effort to identify some of the causal factors contributing these differences, we study the addition of exploratory noise to the agents' selection process, of behavioral diversity between agents, and of narrative momentum in the form of memory of past actions. We make our code available at https://github.com/smearle/picbreeder-vlm.

Large language models (LLMs) are increasingly used to evolve solutions to problems. However, most LLM-evolution frameworks solve static optimization problems, overlooking the open-ended adversarial dynamics that characterize real-world evolutionary processes. Here, we study Digital Red Queen (DRQ), a simple self-play algorithm that embraces these "Red Queen" dynamics via a changing objective. DRQ uses an LLM to evolve assembly programs (warriors) which compete for control of a virtual machine in the game of Core War, a Turing-complete environment studied in artificial life and connected to cybersecurity. In each round of DRQ, the model evolves a new warrior to defeat all previous ones. Over many rounds, warriors become increasingly general (relative to a set of held-out human warriors). Interestingly, across independent runs, we observe a convergence pressure toward a single generalpurpose behavioral strategy, much like convergent evolution in nature. Our work positions Core War as a rich, controllable sandbox for studying adversarial adaptation in artificial systems and for evaluating LLM-based evolution methods. More broadly, the simplicity and effectiveness of DRQ suggest that similarly minimal self-play approaches could prove useful in practical multi-agent adversarial domains, like real-world cybersecurity or combating drug resistance.

Procedural content generation (PCG) enables the creation of vast, varied, and aesthetically rich environments with minimal manual effort. One of the most widely used techniques for procedural map generation is Wave Function Collapse (WFC), a constraint-based algorithm that synthesizes game maps by propagating local patterns while ensuring global consistency. However, despite its effectiveness, WFC often produces repetitive structures and lacks the ability to introduce higher-order spatial coherence or emergent design patterns. This paper explores whether combining Compositional Pattern Producing Networks (CPPN) and WFC - a hybrid method we term CPPN2WFC - leads to more structured and visually compelling game maps compared to using WFC or CPPNs alone. CPPNs, which are artificial neural networks with a selection of different activation functions, have been shown to generate intricate patterns and organic-like structures when evolved through NEAT, a method that dynamically evolves both network topology and weights over generations. By integrating CPPNs into the WFC framework, we introduce an additional layer of flexibility, allowing both constraint satisfaction and high-level structural control. We conduct comparative experiments through an Interactive Evolutionary interface and user study. Main results show that compared to CPPNs or WFC alone, CPPN2WFC strikes a balance between producing global and local patterns.

The ability to continuously and efficiently transfer skills across tasks is a hallmark of biological intelligence and a long-standing goal in artificial systems. Reinforcement learning (RL), a dominant paradigm for learning in high-dimensional control tasks, is known to suffer from brittleness to task variations and catastrophic forgetting. Neuroevolution (NE) has recently gained attention for its robustness, scalability, and capacity to escape local optima. In this paper, we investigate an understudied dimension of NE: its transfer learning capabilities. To this end, we introduce two benchmarks: a) in stepping gates, neural networks are tasked with emulating logic circuits, with designs that emphasize modular repetition and variation b) ecorobot extends the Brax physics engine with objects such as walls and obstacles and the ability to easily switch between different robotic morphologies. Crucial in both benchmarks is the presence of a curriculum that enables evaluating skill transfer across tasks of increasing complexity. Our empirical analysis shows that NE methods vary in their transfer abilities and frequently outperform RL baselines. Our findings support the potential of NE as a foundation for building more adaptable agents and highlight future challenges for scaling NE to complex, real-world problems.

Artificial neural networks can be used to solve a variety of robotic tasks. However, they risk failing catastrophically when faced with out-of-distribution (OOD) situations. Several approaches have employed a type of synaptic plasticity known as Hebbian learning that can dynamically adjust weights based on local neural activities. Research has shown that synaptic plasticity can make policies more robust and help them adapt to unforeseen changes in the environment. However, networks augmented with Hebbian learning can lead to weight divergence, resulting in network instability. Furthermore, such Hebbian networks have not yet been applied to solve legged locomotion in complex real robots with many degrees of freedom. In this work, we improve the Hebbian network with a weight normalization mechanism for preventing weight divergence, analyze the principal components of the Hebbian’s weights, and perform a thorough evaluation of network performance in locomotion control for real 18-DOF dung beetle-like and 16-DOF gecko-like robots. We find that the Hebbian-based plastic network can execute zero-shot sim-to-real adaptation locomotion and generalize to unseen conditions, such as uneven terrain and morphological damage.

Biological brains demonstrate complex neural activity, where neural dynamics are critical to how brains process information. Most artificial neural networks ignore the complexity of individual neurons. We challenge that paradigm. By incorporating neuron-level processing and synchronization, we reintroduce neural timing as a foundational element. We present the Continuous Thought Machine (CTM), a model designed to leverage neural dynamics as its core representation. The CTM has two innovations: (1) neuron-level temporal processing}, where each neuron uses unique weight parameters to process incoming histories; and (2) neural synchronization as a latent representation. The CTM aims to strike a balance between neuron abstractions and biological realism. It operates at a level of abstraction that effectively captures essential temporal dynamics while remaining computationally tractable. We demonstrate the CTM's performance and versatility across a range of tasks, including solving 2D mazes, ImageNet-1K classification, parity computation, and more. Beyond displaying rich internal representations and offering a natural avenue for interpretation owing to its internal process, the CTM is able to perform tasks that require complex sequential reasoning. The CTM can also leverage adaptive compute, where it can stop earlier for simpler tasks, or keep computing when faced with more challenging instances. The goal of this work is to share the CTM and its associated innovations, rather than pushing for new state-of-the-art results. To that end, we believe the CTM represents a significant step toward developing more biologically plausible and powerful artificial intelligence systems. We provide an accompanying [interactive online demonstration](https://pub.sakana.ai/ctm/) and an [extended technical report](https://pub.sakana.ai/ctm/paper).

Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. Their potential to facilitate human coordination with many agents is a promising but largely under-explored area. Such capabilities would be helpful in disaster response, urban planning, and real-time strategy scenarios. In this work, we introduce (1) a real-time strategy game benchmark designed to evaluate these abilities and (2) a novel framework we term HIVE. HIVE empowers a single human to coordinate swarms of up to 2,000 agents through a natural language dialog with an LLM. We present promising results on this multi-agent benchmark, with our hybrid approach solving tasks such as coordinating agent movements, exploiting unit weaknesses, leveraging human annotations, and understanding terrain and strategic points. Our findings also highlight critical limitations of current models, including difficulties in processing spatial visual information and challenges in formulating long-term strategic plans. This work sheds light on the potential and limitations of LLMs in human-swarm coordination, paving the way for future research in this area. The HIVE project page, hive.syrkis.com, includes videos of the system in action.

Artificial intelligence (AI) has enabled agents to master complex video games, from first-person shooters likeCounter-Striketo real-time strategy games such asStarCraft IIand racing games likeGran Turismo. While these achievements are notable, applying these AI methods in commercial video game production remains challenging due to computational constraints. In commercial scenarios, the majority of computational resources are allocated to 3D rendering, leaving limited capacity for AI methods, which often demand high computational power, particularly those relying on pixel-based sensors. Moreover, the gaming industry prioritizes creating human-like behavior in AI agents to enhance player experience, unlike academic models that focus on maximizing game performance. This paper introduces a novel methodology for training neural networks via imitation learning to play a complex, commercial-standard, VALORANT-like 2v2 tactical shooter game, requiring only modest CPU hardware during inference. Our approach leverages an innovative, pixel-free perception architecture using a small set of ray-cast sensors, which capture essential spatial information efficiently. These sensors allow AI to perform competently without the computational overhead of traditional methods. Models are trained to mimic human behavior using supervised learning on human trajectory data, resulting in realistic and engaging AI agents. Human evaluation tests confirm that our AI agents provide human-like gameplay experiences while operating efficiently under computational constraints. This offers a significant advancement in AI model development for tactical shooter games and possibly other genres.

Image-generation AI models have triggered a paradigm shift in how we can express ourselves in visual art. Despite their widespread use in a short amount of time, embedding these models into interactive artifacts is still largely unexplored. In this pictorial, we unpack the design and development process of GenFrame, an image generating picture frame that utilizes generative AI capabilities to mimic traditional paintings. Our work details the necessary steps to integrate generative AI into interactive artifacts and highlights important design considerations for controlling image-generation models in order to achieve specific design intents. Our insights provide interaction designers with a more comprehensive understanding and approach towards utilizing image-generation AI models for interactive artifacts. A demo can be viewed at https://youtu.be/1rhW4fazaBY

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum’s digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum’s collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Artificial neural networks used for reinforcement learning are structurally rigid, meaning that each optimized parameter of the network is tied to its specific placement in the network structure. Structural rigidity limits the ability to optimize parameters of policies across multiple environments that do not share input and output spaces. This is a consequence of the number of optimized parameters being directly dependent on the structure of the network. In this paper, we present Structurally Flexible Neural Networks (SFNNs), which consist of connected gated recurrent units (GRUs) as synaptic plasticity rules and linear layers as neurons. In contrast to earlier work, SFNNs contain several different sets of parameterized building blocks. Here we show that SFNNs can overcome the challenging symmetry dilemma, which refers to the problem of optimizing units with shared parameters to each express different representations during deployment. In this paper, the same SFNN can learn to solve three classic control environments that have different input/output spaces. SFFNs thus represent a step toward a more general model capable of solving several environments at once.

Modular robots are promising for their versatility and large design freedom. Modularity can also enable automatic assembly and reconfiguration, be it autonomous or via external machinery. However, these procedures are error-prone and often result in misassemblings. This, in turn, can cause catastrophic effects on the robot functionality, as the controller deployed in each module is optimized for a different robot shape than the actual one. In this work, we address such shortcoming by proposing a shape-aware modular controller, operating with (1) a self-discovery phase, in which each module controller identifies the shape it is assembled in, followed by (2) a parameter selection phase, where the controller selects its parameters according to the inferred shape. We deploy a self-classifying neural cellular automaton for phase (1), and we leverage evolutionary optimization for implementing a library of controller parameters for phase (2). We test the validity of the proposed method considering voxel-based soft robots, a class of modular soft robots, and the task of locomotion. Our findings confirm the effectiveness of such a controller paradigm, and also show that it can be used to partially overcome unforeseen damages or assembly mistakes.

Biological nervous systems consist of networks of diverse, sophisticated information processors in the form of neurons of different classes. In most artificial neural networks (ANNs), neural computation is abstracted to an activation function that is usually shared between all neurons within a layer or even the whole network; training of ANNs focuses on synaptic optimization. In this paper, we propose the optimization of neuro-centric parameters to attain a set of diverse neurons that can perform complex computations. Demonstrating the promise of the approach, we show that evolving neural parameters alone allows agents to solve various reinforcement learning tasks without optimizing any synaptic weights. While not aiming to be an accurate biological model, parameterizing neurons to a larger degree than the current common practice, allows us to ask questions about the computational abilities afforded by neural diversity in random neural networks. The presented results open up interesting future research directions, such as combining evolved neural diversity with activity-dependent plasticity.

Procedural Content Generation (PCG) is a technique to generate complex and diverse environments in an automated way. However, while generating content with PCG methods is often straightforward, generating meaningful content that reflects specific intentions and constraints remains challenging. Furthermore, many PCG algorithms lack the ability to generate content in an open-ended manner. Recently, Large Language Models (LLMs) have shown to be incredibly effective in many diverse domains. These trained LLMs can be fine-tuned, re-using information and accelerating training for new tasks. Here, we introduce MarioGPT, a fine-tuned GPT2 model trained to generate tile-based game levels, in our case Super Mario Bros levels. MarioGPT can not only generate diverse levels, but can be text-prompted for controllable level generation, addressing one of the key challenges of current PCG techniques. As far as we know, MarioGPT is the first text-to-level model and combined with novelty search it enables the generation of diverse levels with varying play-style dynamics (i.e. player paths) and the open-ended discovery of an increasingly diverse range of content. Code available at https://github.com/shyamsn97/mario-gpt.

Generative adversarial networks (GANs) are a powerful indirect genotype-to-phenotype mapping for evolutionary search. Much previous work applying GANs to level generation focuses on fixed-size segments combined into a whole level, but individual segments may not fit together cohesively. In contrast, segments in human designed levels are often repeated, directly or with variation, and organized into patterns (the symmetric eagle in Level 1 ofThe Legend of Zelda, or repeated pipe motifs inSuper Mario Bros.). Such patterns can be produced with compositional pattern producing networks (CPPNs). CPPNs define latent vector GAN inputs as a function of geometry, organizing segments output by a GAN into complete levels. However, collections of latent vectors can also be evolved directly, producing more chaotic levels. We propose a hybrid approach that evolves CPPNs first, but allows latent vectors to evolve later, combining the benefits of both approaches. These approaches are evaluated inSuper Mario Bros. andThe Legend of Zelda. We previously demonstrated via a quality-diversity algorithm that CPPNs better cover the space of possible levels than directly evolved levels. Here, we show that the hybrid approach first, covers areas that neither of the other methods can, and second, achieves comparable or superior quality diversity (QD) scores.

Deep generative models can automatically create content of diverse types. However, there are no guarantees that such content will satisfy the criteria necessary to present it to end-users and be functional, e.g. the generated levels could be unsolvable or incoherent. In this paper we study this problem from a geometric perspective, and provide a method for reliable interpolation and random walks in the latent spaces of Categorical VAEs based on Riemannian geometry. We test our method with “Super Mario Bros” and “The Legend of Zelda” levels, and against simpler baselines inspired by current practice. Results show that the geometry we propose is better able to interpolate and sample, reliably staying closer to parts of the latent space that decode to playable content.

Organisms in nature have evolved to exhibit flexibility in face of changes to the environment and/or to themselves. Artificial neural networks (ANNs) have proven useful for controlling of artificial agents acting in environments. However, most ANN models used for reinforcement learning-type tasks have a rigid structure that does not allow for varying input sizes. Further, they fail catastrophically if inputs are presented in an ordering unseen during optimization. We find that these two ANN inflexibilities can be mitigated and their solutions are simple and highly related. For permutation invariance, no optimized parameters can be tied to a specific index of the input elements. For size invariance, inputs must be projected onto a common space that does not grow with the number of projections. Based on these restrictions, we construct a conceptually simple model that exhibit flexibility most ANNs lack. We demonstrate the model's properties on multiple control problems, and show that it can cope with even very rapid permutations of input indices, as well as changes in input size. Ablation studies show that is possible to achieve these properties with simple feedforward structures, but that it is much easier to optimize recurrent structures.

Materials with the ability to self-classify their own shape have the potential to advance a wide range of engineering applications and industries. Biological systems possess the ability not only to self-reconfigure but also to self-classify themselves to determine a general shape and function. Previous work into modular robotics systems has only enabled self-recognition and self-reconfiguration into a specific target shape, missing the inherent robustness present in nature to self-classify. In this paper we therefore take advantage of recent advances in deep learning and neural cellular automata, and present a simple modular 2D robotic system that can infer its own class of shape through the local communication of its components. Furthermore, we show that our system can be successfully transferred to hardware which thus opens op-portunities for future self-classifying machines. Code available at https://github.com/kattwalker/projectcube. Video available at https://youtu.be/0TCOkE4keyc.

The rise of digital technologies and Machine Learning (ML)-tools for creative expression brings about novel opportunities for studying creativity and cognition at scale. In this paper, we present a pilot study of crea.blender SDG - an online GAN based image generation game. We designed crea.blender SDG with two goals in mind: The first, to let people create images relating to the United Nations Sustainable Development Goals (SDGs) and through them, engage in large-scale conversations on complex socioscientific problems. The second, as a fun and inspiring gateway for public participation in research, generating data for the creativity and cognition research and design community. Specifically in this pilot, we study and affirm that the design of crea.blender SDG is flexible enough to allow users to create images that express both anxiety and hope for the future; affirm that user generated images express these ideas in ways that are meaningful to people other than the original creator; and begin to investigate which specific features of images are more closely related to dystopian or utopian ideas of the future. Finally, we discuss implications for future design and research with ML-based creativity tools.

The advent of artificial intelligence (AI) and machine learning (ML) bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and experimenting with how humans interact with AI. Through a systematic survey of neural network games (n = 38), we identified the dominant interaction metaphors and AI interaction patterns in these games. In addition, we applied existing human-AI interaction guidelines to further shed light on player-AI interaction in the context of AI-infused systems. Our core finding is that AI as play can expand current notions of human-AI interaction, which are predominantly productivity-based. In particular, our work suggests that game and UX designers should consider flow to structure the learning curve of human-AI interaction, incorporate discovery-based learning to play around with the AI and observe the consequences, and offer users an invitation to play to explore new forms of human-AI interaction.

In games, as well as many user-facing systems, adapting content to users' preferences and experience is an important challenge. This paper explores a novel method to realize this goal in the context of dynamic difficulty adjustment (DDA). Here the aim is to constantly adapt the content of a game to the skill level of the player, keeping them engaged by avoiding states that are either too difficult or too easy. Current systems for DDA rely on expensive data mining, or on handcrafted rules designed for particular domains, and usually adapts to keep players in the flow, leaving no room for the designer to present content that is purposefully easy or difficult. This paper presents Fast Bayesian Content Adaption (FBCA), a system for DDA that is agnostic to the domain and that can target particular difficulties. We deploy this framework in two different domains: the puzzle game Sudoku, and a simple Roguelike game. By modifying the acquisition function's optimization, we are reliably able to present a content with a bespoke difficulty for players with different skill levels in less than five iterations for Sudoku and fifteen iterations for the simple Roguelike. Our method significantly outperforms simpler DDA heuristics with the added benefit of maintaining a model of the user. These results point towards a promising alternative for content adaption in a variety of different domains.

This paper presents a Mixed-Initiative version of Squeezer, a tool for designing juice effects in the Unity game engine. Drawing upon sound synthesizers and game description languages, Squeezer can synthesize common types of juice effects by combining simple building blocks into sequences. Additionally, Squeezer offers effect generation based on predefined recipes as well as an interface for interactively evolving effect sequences. We conducted a user study with five experts to verify the functionality and interest among game designers. By applying generative and evolutionary strategies to juice effect design, Squeezer allows game designers and researchers using games in their work to explore adding juice effects to their games and frameworks.

Applying neural network (NN) methods in games can lead to various new and exciting game dynamics not previously possible. However, they also lead to new challenges such as the lack of large, clean datasets, varying player skill levels, and changing gameplay strategies. In this paper, we focus on the adversarial player strategy aspect in the game iNNk, in which players try to communicate secret code words through drawings with the goal of not being deciphered by a NN. Some strategies exploit weaknesses in the NN that consistently trick it into making incorrect classifications, leading to unbalanced gameplay. We present a method that combines transfer learning and ensemble methods to obtain a data-efficient adaptation to these strategies. This combination significantly outperforms the baseline NN across all adversarial player strategies despite only being trained on a limited set of adversarial examples. We expect the methods developed in this paper to be useful for the rapidly growing field of NN-based games, which will require new approaches to deal with unforeseen player creativity.

Generalization to out-of-distribution (OOD) circumstances after training remains a challenge for artificial agents. To improve the robustness displayed by plastic Hebbian neural networks, we evolve a set of Hebbian learning rules, where multiple connections are assigned to a single rule. Inspired by the biological phenomenon of the genomic bottleneck, we show that by allowing multiple connections in the network to share the same local learning rule, it is possible to drastically reduce the number of trainable parameters, while obtaining a more robust agent. During evolution, by iteratively using simple K-Means clustering to combine rules, our Evolve & Merge approach is able to reduce the number of trainable parameters from 61,440 to 1,920, while at the same time improving robustness, all without increasing the number of generations used. While optimization of the agents is done on a standard quadruped robot morphology, we evaluate the agents' performances on slight morphology modifications in a total of 30 unseen morphologies. Our results add to the discussion on generalization, overfitting and OOD adaptation. To create agents that can adapt to a wider array of unexpected situations, Hebbian learning combined with a regularising "genomic bottleneck" could be a promising research direction.

Despite recent advances in object detection using deep learning neural networks, these neural networks still struggle to identify objects in art images such as paintings and drawings. This challenge is known as the cross depiction problem and it stems in part from the tendency of neural networks to prioritize identification of an object's texture over its shape. In this paper we propose and evaluate a process for training neural networks to localize objects — specifically people — in art images. We generate a large dataset for training and validation by modifying the images in the COCO dataset using AdaIn style transfer. This dataset is used to fine-tune a Faster R-CNN object detection network, which is then tested on the existing People-Art testing dataset. The result is a significant improvement on the state of the art and a new way forward for creating datasets to train neural networks to process art images.

We introduce a novel combination of Bayesian Models (BMs) and Neural Networks (NNs) for making predictions with a minimum expected risk. Our approach combines the best of both worlds, the data efficiency and interpretability of a BM with the speed of a NN. For a BM, making predictions with the lowest expected loss requires integrating over the posterior distribution. When exact inference of the posterior predictive distribution is intractable, approximation methods are typically applied, e.g. Monte Carlo (MC) simulation. For MC, the variance of the estimator decreases with the number of samples – but at the expense of increased computational cost. Our approach removes the need for iterative MC simulation on the CPU at prediction time. In brief, it works by fitting a NN to synthetic data generated using the BM. In a single feed-forward pass, the NN gives a set of point-wise approximations to the BM's posterior predictive distribution for a given observation. We achieve risk minimized predictions significantly faster than standard methods with a negligible loss on the test dataset. We combine this approach with Active Learning (AL) to minimize the amount of data required for fitting the NN. This is done by iteratively labeling more data in regions with high predictive uncertainty of the NN.

Facial composites are graphical representations of an eyewitness's memory of a face. Many digital systems are available for the creation of such composites but are either unable to reproduce features unless previously designed or do not allow holistic changes to the image. In this paper, we improve the efficiency of composite creation by removing the reliance on expert knowledge and letting the system learn to represent faces from examples. The novel approach, Composite Generating GAN (CG-GAN), applies generative and evolutionary computation to allow casual users to easily create facial composites. Specifically, CG-GAN utilizes the generator network of a pg-GAN to create high-resolution human faces. Users are provided with several functions to interactively breed and edit faces. CG-GAN offers a novel way of generating and handling static and animated photo-realistic facial composites, with the possibility of combining multiple representations of the same perpetrator, generated by different eyewitnesses.

Methods for dynamic difficulty adjustment allow games to be tailored to particular players to maximize their engagement. However, current methods often only modify a limited set of game features such as the difficulty of the opponents, or the availability of resources. Other approaches, such as experience-driven Procedural Content Generation (PCG), can generate complete levels with desired properties such as levels that are neither too hard nor too easy, but require many iterations. This paper presents a method that can generate and search for complete levels with a specific target difficulty in only a few trials. This advance is enabled by through an Intelligent Trial-and-Error algorithm, originally developed to allow robots to adapt quickly. Our algorithm first creates a large variety of different levels that vary across predefined dimensions such as leniency or map coverage. The performance of an AI playing agent on these maps gives a proxy for how difficult the level would be for another AI agent (e.g. one that employs Monte Carlo Tree Search instead of Greedy Tree Search); using this information, a Bayesian Optimization procedure is deployed, updating the difficulty of the prior map to reflect the ability of the agent. The approach can reliably find levels with a specific target difficulty for a variety of planning agents in only a few trials, while maintaining an understanding of their skill landscape.

Imitation Learning (IL) is a machine learning approach to learn a policy from a set of demonstrations. IL can be useful to kick-start learning before applying reinforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. Despite the success of systems that use IL and RL, how such systems can adapt in-between game rounds is a neglected area of study but an important aspect of many strategy games. In this paper, we present a new approach called Behavioral Repertoire Imitation Learning (BRIL) that learns a repertoire of behaviors from a set of demonstrations by augmenting the state-action pairs with behavioral descriptions. The outcome of this approach is a single neural network policy conditioned on a behavior description that can be precisely modulated. We apply this approach to train a policy on 7,777 human demonstrations for the build-order planning task in StarCraft II. Dimensionality reduction is applied to construct a low-dimensional behavioral space from a high-dimensional description of the army unit composition of each human replay. The results demonstrate that the learned policy can be effectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, the policy can adapt its behavior - in-between games - to reach a performance beyond that of the traditional IL baseline approach.

Generative Adversarial Networks (GANs) have shown impressive results for image generation. However, GANs face challenges in generating contents with certain types of constraints, such as game levels. Specifically, it is difficult to generate levels that have aesthetic appeal and are playable at the same time. Additionally, because training data usually is limited, it is challenging to generate unique levels with current GANs. In this paper, we propose a new GAN architecture named Conditional Embedding Self-Attention Generative Adversarial Net-work (CESAGAN) and a new bootstrapping training procedure. The CESAGAN is a modification of the self-attention GAN that incorporates an embedding feature vector input to condition the training of the discriminator and generator. This allows the network to model non-local dependency between game objects, and to count objects. Additionally, to reduce the number of levels necessary to train the GAN, we propose a bootstrapping mechanism in which playable generated levels are added to the training set. The results demonstrate that the new approach does not only generate a larger number of levels that are playable but also generates fewer duplicate levels compared to a standard GAN.

Recent procedural content generation via machine learning (PCGML) methods allow learning from existing content to produce similar content automatically. While these approaches are able to generate content for different games (e.g. Super Mario Bros., DOOM, Zelda, and Kid Icarus), it is an open question how well these approaches can capture large-scale visual patterns such as symmetry. In this paper, we propose match-three games as a domain to test PCGML algorithms regarding their ability to generate suitable patterns. We demonstrate that popular algorithms such as Generative Adversarial Networks struggle in this domain and propose adaptations to improve their performance. In particular, we augment the neighbourhood of a Markov Random Fields approach to take into account not only local but also symmetric positional information. We conduct several empirical tests, including a user study that show the improvements achieved by the proposed modifications and obtain promising results.

Generative Adversarial Networks (GANs) are an emerging form of indirect encoding. The GAN is trained to induce a latent space on training data, and a real-valued evolutionary algorithm can search that latent space. Such Latent Variable Evolution (LVE) has recently been applied to game levels. However, it is hard for objective scores to capture level features that are appealing to players. Therefore, this paper introduces a tool for interactive LVE of tile-based levels for games. The tool also allows for direct exploration of the latent dimensions, and allows users to play discovered levels. The tool works for a variety of GAN models trained for both Super Mario Bros. and The Legend of Zelda, and is easily generalizable to other games. A user study shows that both the evolution and latent space exploration features are appreciated, with a slight preference for direct exploration, but combining these features allows users to discover even better levels. User feedback also indicates how this system could eventually grow into a commercial design tool, with the addition of a few enhancements.

Generative Adversarial Networks (GANs) are proving to be a powerful indirect genotype-to-phenotype mapping for evolutionary search, but they have limitations. In particular, GAN output does not scale to arbitrary dimensions, and there is no obvious way of combining multiple GAN outputs into a cohesive whole, which would be useful in many areas, such as the generation of video game levels. Game levels often consist of several segments, sometimes repeated directly or with variation, organized into an engaging pattern. Such patterns can be produced with Compositional Pattern Producing Networks (CPPNs). Specifically, a CPPN can define latent vector GAN inputs as a function of geometry, which provides a way to organize level segments output by a GAN into a complete level. This new CPPN2GAN approach is validated in both Super Mario Bros. and The Legend of Zelda. Specifically, divergent search via MAP-Elites demonstrates that CPPN2GAN can better cover the space of possible levels. The layouts of the resulting levels are also more cohesive and aesthetically consistent.

Lifelong learning and adaptability are two defining aspects of biological agents. Modern reinforcement learning (RL) approaches have shown significant progress in solving complex tasks, however once training is concluded, the found solutions are typically static and incapable of adapting to new information or perturbations. While it is still not completely understood how biological brains learn and adapt so efficiently from experience, it is believed that synaptic plasticity plays a prominent role in this process. Inspired by this biological mechanism, we propose a search method that, instead of optimizing the weight parameters of neural networks directly, only searches for synapse-specific Hebbian learning rules that allow the network to continuously self-organize its weights during the lifetime of the agent. We demonstrate our approach on several reinforcement learning tasks with different sensory modalities and more than 450K trainable plasticity parameters. We find that starting from completely random weights, the discovered Hebbian rules enable an agent to navigate a dynamical 2D-pixel environment; likewise they allow a simulated 3D quadrupedal robot to learn how to walk while adapting to morphological damage not seen during training and in the absence of any explicit reward or error signal in less than 100 timesteps.

Many factors influence the evolvability of populations, and this article illustrates how intrinsic mortality (death induced through internal factors) in an evolving population contributes favorably to evolvability on a fixed deceptive fitness landscape. We test for evolvability using the hierarchical if-and-only-if (h-iff) function as a deceptive fitness landscape together with a steady state genetic algorithm (SSGA) with a variable mutation rate and indiscriminate intrinsic mortality rate. The mutation rate and the intrinsic mortality rate display a relationship for finding the global maximum. This relationship was also found when implementing the same deceptive fitness landscape in a spatial model consisting of an evolving population. We also compared the performance of the optimal mutation and mortality rate with a state-of-the-art evolutionary algorithm called age-fitness Pareto optimization (AFPO) and show how the two approaches traverse the h-iff landscape differently. Our results indicate that the intrinsic mortality rate and mutation rate induce random genetic drift that allows a population to efficiently traverse a deceptive fitness landscape. This article gives an overview of how intrinsic mortality influences the evolvability of a population. It thereby supports the premise that programmed death of individuals could have a beneficial effect on the evolvability of the entire population.

In this paper, we review recent deep learning advances in the context of how they have been applied to play different types of video games such as first-person shooters, arcade games, and real-time strategy games. We analyze the unique requirements that different game genres pose to a deep learning system and highlight important open challenges in the context of applying these machine learning methods to video games, such as general game playing, dealing with extremely large decision spaces and sparse rewards.

This paper introduces UnityVGDL, a port of the Video Game Description Language (VGDL) to the widely used Unity game engine. Our framework is based on the General Video Game AI (GVGAI) competition framework and implements its core ontology, including a forward model. It integrates the Unity Machine Learning Agents (ML-Agents) toolkit with VGDL to train and run agents in VGDL-described games. We compare baseline learning results between GVGAI and UnityVGDL across four different games and conclude that the Unity port is comparable to the GVGAI framework. UnityVGDL is available at: https://github.com/pyjamads/UnityVGDL

From an early point, games have been promoted as important challenges within the research field of Artificial Intelligence (AI). Recent developments in machine learning have allowed a few AI systems to win against top professionals in even the most challenging video games, including Dota 2 and StarCraft. It thus may seem that AI has now achieved all of the long-standing goals that were set forth by the research community. In this paper, we introduce a black box approach that provides a pragmatic way of evaluating the fairness of AI vs. human competitions, by only considering motoric and perceptual fairness on the competitors’ side. Additionally, we introduce the notion of extrinsic and intrinsic factors of a game competition and apply these to discuss and compare the competitions in relation to human vs. human competitions. We conclude that Dota 2 and StarCraft II are not yet mastered by AI as they so far only have been able to win against top professionals in limited competition structures in restricted variants of the games.

We propose the popular board game Blood Bowl as a new challenge for Artificial Intelligence (AI). Blood Bowl is a fully-observable, stochastic, turn-based, modern-style board game with a grid-based game board. At first sight, the game ought to be approachable by numerous game-playing algorithms. However, as all pieces on the board belonging to a player can be moved several times each turn, the turn-wise branching factor becomes overwhelming for traditional algorithms. Additionally, scoring points in the game is rare and difficult, which makes it hard to design heuristics for search algorithms or apply reinforcement learning. We present the Fantasy Football AI (FFAI) framework that implements the core rules of Blood Bowl and includes a forward model, several OpenAI Gym environments for reinforcement learning, competition functionalities, and a web application that allows for human play. We also present Bot Bowl I, the first AI competition that will use FFAI along with baseline agents and preliminary reinforcement learning results. Additionally, we present a wealth of opportunities for future AI competitions based on FFAI.

Neural architectures inspired by our own human cognitive system, such as the recently introduced world models, have been shown to outperform traditional deep reinforcement learning (RL) methods in a variety of different domains. Instead of the relatively simple architectures employed in most RL experiments, world models rely on multiple different neural components that are responsible for visual information processing, memory, and decision-making. However, so far the components of these models have to be trained separately and through a variety of specialized training methods. This paper demonstrates the surprising finding that models with the same precise parts can be instead efficiently trained end-to-end through a genetic algorithm (GA), reaching a comparable performance to the original world model by solving a challenging car racing task. An analysis of the evolved visual and memory system indicates that they include a similar effective representation to the system trained through gradient descent. Additionally, in contrast to gradient descent methods that struggle with discrete variables, GAs also work directly with such representations, opening up opportunities for classical planning in latent space. This paper adds additional evidence on the effectiveness of deep neuroevolution for tasks that require the intricate orchestration of multiple components in complex heterogeneous architectures.

Soft robotics is a growing field of research and one of its challenges is how to efficiently design a controller for a soft morphology. This paper presents a marine soft robot inspired by the ghost knifefish that swims on the water surface by using an undulating fin underneath its body. We investigate how propagating wave functions can be evolved and how these affect the swimming performance of the robot. The fin and body of the robot are constructed from silicone and six wooden fin rays actuated by servo motors. In order to bypass the reality gap, which would necessitate a complex simulation of the fish, we implemented a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) directly on the physical robot to optimize its controller for travel speed. Our results show that evolving a simple sine wave or a Fourier series can generate controllers that outperform a hand programmed controller. The results additionally demonstrate that the best evolved controllers share similarities with the undulation patterns of actual knifefish. Based on these results we suggest that evolution on physical robots is promising for future application in optimizing behaviors of soft robots.

Generative Adversarial Networks (GANs) are a machine learning approach capable of generating novel example outputs across a space of provided training examples. Procedural Content Generation (PCG) of levels for video games could benefit from such models, especially for games where there is a pre-existing corpus of levels to emulate. This paper trains a GAN to generate levels for Super Mario Bros using a level from the Video Game Level Corpus. The approach successfully generates a variety of levels similar to one in the original corpus, but is further improved by application of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). Specifically, various fitness functions are used to discover levels within the latent space of the GAN that maximize desired properties. Simple static properties are optimized, such as a given distribution of tile types. Additionally, the champion A* agent from the 2009 Mario AI competition is used to assess whether a level is playable, and how many jumping actions are required to beat it. These fitness functions allow for the discovery of levels that exist within the space of examples designed by experts, and also guide the search towards levels that fulfill one or more specified objectives.

Bio-hybrid systems-close couplings of natural organisms with technology-are high potential and still underexplored. In existing work, robots have mostly influenced group behaviors of animals. We explore the possibilities of mixing robots with natural plants, merging useful attributes. Significant synergies arise by combining the plants' ability to efficiently produce shaped material and the robots' ability to extend sensing and decision-making behaviors. However, programming robots to control plant motion and shape requires good knowledge of complex plant behaviors. Therefore, we use machine learning to create a holistic plant model and evolve robot controllers. As a benchmark task we choose obstacle avoidance. We use computer vision to construct a model of plant stem stiffening and motion dynamics by training an LSTM network. The LSTM network acts as a forward model predicting change in the plant, driving the evolution of neural network robot controllers. The evolved controllers augment the plants' natural light-finding and tissue-stiffening behaviors to avoid obstacles and grow desired shapes. We successfully verify the robot controllers and bio-hybrid behavior in reality, with a physical setup and actual plants.

We address the problem of playing turn-based multiaction adversarial games, which include many strategy games with extremely high branching factors as players take multiple actions each turn. This leads to the breakdown of standard tree search methods, including Monte Carlo tree search (MCTS), as they become unable to reach a sufficient depth in the game tree. In this paper, we introduce online evolutionary planning (OEP) to address this challenge, which searches for combinations of actions to perform during a single turn guided by a fitness function that evaluates the quality of a particular state. We compare OEP to different MCTS variations that constrain the exploration to deal with the high branching factor in the turn-based multiaction game Hero Academy. While the constrained MCTS variations outperform the vanilla MCTS implementation by a large margin, OEP is able to search the space of plans more efficiently than any of the tested tree search methods as it has a relative advantage when the number of actions per turn increases.

The real-time strategy game StarCraft has become an important benchmark for AI research as it poses a complex environment with numerous challenges. An important strategic aspect in this game is to decide what buildings and units to produce. StarCraft bots playing in AI competitions today are only able to switch between predefined strategies, which makes it hard to adapt to new situations. This paper introduces an evolutionary-based method to overcome this challenge, called Continual Online Evolutionary Planning (COEP), which is able to perform in-game adaptive build-order planning. COEP was added to an open source StarCraft bot called UAlbertaBot and is able to outperform the built-in bots in the game as well as being competitive against a number of scripted opening strategies. The COEP augmented bot can change its build order dynamically and quickly adapt to the opponent's strategy.

This paper describes an evaluation conducted on the MetaCompose music generator, which is based on evolutionary computation and uses a hybrid evolutionary technique that combines FI-2POP and multi-objective optimization. The main objective of MetaCompose is to create music in real-time that can express different mood-states. The experiment presented here aims to evaluate: (i) if the perceived mood experienced by the participants of a music score matches intended mood the system is trying to express and (ii) if participants can identify transitions in the mood expression that occur mid-piece. Music clips including transitions and with static affective states were produced by MetaCompose and a quantitative user study was performed. Participants were tasked with annotating the perceived mood and moreover were asked to annotate in real-time changes in valence. The data collected confirms the hypothesis that people can recognize changes in music mood and that MetaCompose can express perceptibly different levels of arousal. In regards to valence we observe that, while it is mainly perceived as expected, changes in arousal seems to also influence perceived valence, suggesting that one or more of the music features MetaCompose associates with arousal has some effect on valence as well.

Evolutionary robotics is challenged with some key problems that must be solved, or at least mitigated extensively, before it can fulfill some of its promises to deliver highly autonomous and adaptive robots. The reality gap and the ability to transfer phenotypes from simulation to reality constitute one such problem. Another lies in the embodiment of the evolutionary processes, which links to the first, but focuses on how evolution can act on real agents and occur independently from simulation, that is, going from being, as Eiben, Kernbach, & Haasdijk [2012, p. 261] put it, "the evolution of things, rather than just the evolution of digital objects.…" The work presented here investigates how fully autonomous evolution of robot controllers can be realized in hardware, using an industrial robot and a marker-based computer vision system. In particular, this article presents an approach to automate the reconfiguration of the test environment and shows that it is possible, for the first time, to incrementally evolve a neural robot controller for different obstacle avoidance tasks with no human intervention. Importantly, the system offers a high level of robustness and precision that could potentially open up the range of problems amenable to embodied evolution.

Neuroevolution [i.e., evolving artificial neural networks (ANNs) through evolutionary algorithms] has shown promise in evolving agents and robot controllers, which display complex behaviors and can adapt to their environments. These properties are also relevant to video games, since they can increase their longevity and replayability. However, the design of most current games precludes the use of any techniques which might yield unpredictable or even open-ended results. This paper describes the game EvoCommander, with the goal to further demonstrate the potential of neuroevolution in games. In EvoCommander the player incrementally evolves an arsenal of ANN-controlled behaviors (e.g., ranged attack, flee, etc.) for a simple robot that has to battle other player and computer controlled robots. The game introduces the novel game mechanic of “brain switching,” selecting which evolved neural network is active at any point during battle. Results from playtests indicate that brain switching is a promising new game mechanic, leading to players employing interesting different strategies when training their robots and when controlling them in battle.

This paper surveys research on applying neuroevolution (NE) to games. In neuroevolution, artificial neural networks are trained through evolutionary algorithms, taking inspiration from the way biological brains evolved. We analyze the application of NE in games along five different axes, which are the role NE is chosen to play in a game, the different types of neural networks used, the way these networks are evolved, how the fitness is determined and what type of input the network receives. The paper also highlights important open research challenges in the field.

Due to the replacement of natural flora and fauna with ur- ban environments, a significant part of the earth’s organisms that function as primary consumers have been dispelled. To compensate for the reduction in the amount of primary con- sumers, robotic systems that mimic plant-like organisms are interesting to mimic for their potential functional and aes- thetic value in urban environments. To investigate how to utilize plant developmental strategies in order to engender ur- ban artificial plants, we built a simple evolutionary model that applies an L-System based grammar as an abstraction of plant development. In the presented experiments, phytomorpholo- gies (plant morphologies) are iteratively constructed using a context sensitive L-System. The genomic representation of the L-System is subject to mutation by an evolutionary al- gorithm. These mutations thus alter the developmental rules of these phytomorphologies. We compare the differences be- tween the light absorption of evolving virtual plants that re- main static during their life and virtual plants that possess the possibility to move joints that link the separate parts of the virtual plants. Our results show that our evolutionary al- gorithm did not exploit potential beneficial joint actuation, instead, mostly static structures evolved. The results of our evolving L-System show that it is able to create various phy- tomorphologies, albeit that the results are preliminary and will be more thoroughly investigated in the future

An unsolved problem in neuroevolution (NE) is to evolve artificial neural networks (ANN) that can store and use information to change their behavior online. While plastic neural networks have shown promise in this context, they have difficulties retaining information over longer periods of time and integrating new information without losing previously acquired skills. Here we build on recent work by Graves et al. [5] who extended the capabilities of an ANN by combining it with an external memory bank trained through gradient descent. In this paper, we introduce an evolvable version of their Neural Turing Machine (NTM) and show that such an approach greatly simplifies the neural model, generalizes better, and does not require accessing the entire memory content at each time-step. The Evolvable Neural Turing Machine (ENTM) is able to solve a simple copy tasks and for the first time, the continuous version of the double T-Maze, a complex reinforcement-like learning problem. In the T-Maze learning task the agent uses the memory bank to display adaptive behavior that normally requires a plastic ANN, thereby suggesting a complementary and effective mechanism for adaptive behavior in NE.

An open problem in neuroevolution (i.e. evolving artificial neural networks) is to evolve complex cognitive behaviors that allow robots to adapt and learn from past experience. While previous studies on the evolution of cognitive behaviors have shown that more explorative search methods such as novelty search, outperform traditional objective-based approaches, evolving more sophisticated cognitive capabilities remains difficult. In this context, a major challenge is the deceptive nature of learning to learn. Because it is easier at first to improve fitness without evolving the ability to learn, evolution often converges on non-adaptive solutions. The novel hypothesis in this paper is that we can leverage human insights during the search for cognitive behaviors because of our ability to more easily distinguish between adaptive and non-adaptive solutions than novelty or fitness-based approaches. This paper shows that the recently introduced method novelty-assisted interactive evolution (NA-IEC), which combines human intuition with novelty search, allows the evolution of cognitive behaviors in a T-Maze domain faster than fully-automated searches by themselves.

This special issue displays some of the best articles presented at the Fourteenth International Conference on the Synthesis and Simulation of Living Systems (ALife 14), which was held in New York, USA, on July 30–August 2, 2014 (http://alife2014.alife.org/). ALife 14 was the fourteenth convening of ALife, one of the two largest conference series focusing on the study of artificial life, alternating with ECAL, the European edition.ALife 14 attracted a total of 203 submissions, solicited in two formats: full article (up to 8 pages) and extended abstract (2 pages). Both types were evaluated through multiple peer reviews by the Program Committee. As a result, 102 submissions were accepted for oral presentations and 62 for poster presentations. These articles and abstracts were published in the open-access conference proceedings available from MIT Press' website (https://mitpress.mit.edu/index.php?q=books/artificial-life-14). The conference program was full of scientific, professional, and creative activities, including five superb keynote talks, twenty parallel sessions covering a wide variety of topical areas, a very well-attended poster session with a plenary “Poster Blitz Movie Show,” nine workshops, six tutorials, the First Summer School of the International Society for Artificial Life (ISAL) with four lectures, a science visualization competition, and a career-advising luncheon for postdocs and graduate students. These were four vibrant, intellectually stimulating days in Manhattan.The contributions included in this special issue were selected, by the Organizing Committee and the Best Paper Committee, from the full articles presented orally at the ALife 14 conference, based on their peer review scores and the quality of their presentations. The Best Paper and the Best Student Paper of ALife 14 (http://blogs.cornell.edu/alife14nyc/best-paperposter-awards/) were also included in this special issue. Authors submitted an extended version of their conference manuscript, which went through another round of peer review and revision for journal publication. As a result, there are eight articles included in this special issue. The first four are about artificial chemistry: two discussing abstract models and two addressing real biochemistry. The last four articles are about evolution, two of them also covering morphological adaptation and computing. Their contents are summarized below.The first article, “Complex Autocatalysis in Simple Chemistries,” received the Best Paper award at ALife 14. In this article, Virgo, Ikegami, and McGregor present simple mass-action-kinetics-based artificial chemistry models and demonstrate that, even in a thermodynamically reversible chemical reaction network, complex autocatalytic cycles can emerge when certain direct reactions are prohibited. Complex nonlinear dynamics, such as bistability, arising in their simple models may offer new insight into the origins of life.In “Exploring the Space of Viable Configurations in a Model of Metabolism-Boundary Co-construction,” Agmon, Gates, Churavy, and Beer propose a spatial artificial chemistry model in which autopoietic interaction between boundary formation and metabolic reactions inside it allows stable cellular structures to emerge. They systematically examine the robustness and variability of these cellular structures by subjecting them to global or local perturbations, revealing a transition network of viable configurations.In “Computational Design of a Circular RNA with Prionlike Behavior,” Badelt, Flamm, and Hofacker present a novel theoretical chemistry method to computationally design RNA molecules that have desired properties (energy landscapes), using their software tool called the Vienna RNA package. They demonstrate the effectiveness of the proposed method by using it to design self-replicating RNA molecules that show conformational self-replication, just like the self-replication of prions.In “Compartmentalization of an all-E. coli Cell-Free Expression System for the Construction of a Minimal Cell,” Caschera and Noireaux present an experimental study on the effects of compartmentalization of their cell-free transcription-translation system extracted from E. coli. They show that, when encapsulated in cell-sized lipid vesicles, this system exhibits large fluctuations in gene expression levels, possibly due to heterogeneity in DNA template encapsulation.The next article, “An Informational Study of the Evolution of Codes and of Emerging Concepts in Populations of Agents,” was the recipient of the Best Student Paper award at ALife 14. In this article, Burgos and Polani study an agent population model in which no shared code is initially available for agents' communication and the sources of information are not distinguishable to each agent. Using evolutionary optimization, they demonstrate that a universal code can emerge if the population is well mixed, while multiple distinct codes may result in spatially structured populations. Moreover, “blind” agents can develop concepts about the environment using information coming from other agents.In “A General Statistical Method for Identifying Adaptations by Parameterizing Trait Space,” Blount proposes a formal method to detect adaptation by mapping the traits of evolving organisms to a 3D metric space that consists of variation, heritability, and differential fitness. This method does not require a priori specification of what kind of function or purpose the adaptation possesses. As a proof of concept, the calculation of differential fitness is demonstrated through the application of the proposed method to Packard's Bugs system.In “Population and Evolutionary Dynamics based on Predator-Prey Relationships in a 3D Physical Simulation,” Ito, Pilat, Suzuki, and Arita present a virtual ecosystem of 3D morphological creatures like Sims' blockies, where both ecological and evolutionary dynamics develop simultaneously and spontaneously via coevolution of predator and prey populations. Computer simulations show the emergence of short-term (ecological) and long-term (evolutionary) dynamics occurring in two different regimes of time scales, as well as their mutual interactions.Finally, in “Active Shape Discrimination with Compliant Bodies as Reservoir Computers,” Johnson, Philippides, and Husbands propose to use a randomly generated mass-spring-damper (MSD) network embedded in an agent's body as a “morphogenetic computation” device, which is a physical counterpart of neural-network-based reservoir computing. They demonstrate, through evolutionary search, that such an MSD network can be used successfully as a reservoir computer to control the agent when catching objects of a certain shape while avoiding others.We thank all the authors of the above articles for their excellent contributions, which nicely illustrate the current state of the art of artificial life. We would also like to thank the following people on the Best Paper Committee of ALife 14, who played a crucial role in evaluating numerous articles and selecting the best ones to be included in this volume:• Chair: Luis Rocha• Co-chair: Takashi Ikegami• Committee members: Joshua Auerbach, Lola Cañamero, Dominique Chu, Sylvain Cussat-Blanc, Stéphane Doncieux, Dusan Misevic, Susan Stepney, Sebastian von MammenThe success of the ALife 14 conference was due to all who helped and participated in this event, including the Program Committee members, the five keynote speakers (John H. Conway, Lee Cronin, Naomi Leonard, Jesse Louis-Rosenberg, and Karl Sims), the organizers of workshops and tutorials, local organization staff, and the following sponsors:• US National Science Foundation• MIT Press• Cornell University• Thomas J. Watson School of Engineering and Applied Science at Binghamton University• Wolfram Research• Mary Ann Liebert, Inc. PublishersFinally, we would like to give special thanks to Mark Bedau, the Editor-in-Chief of the Artificial Life journal, and Linda Reedijk, the editorial assistant, for their help, encouragement, and support in this special issue.

We present a survey of the first 21 years of web-based artificial life (WebAL) research and applications, broadly construed to include the many different ways in which artificial life and web technologies might intersect. Our survey covers the period from 1994-when the first WebAL work appeared-up to the present day, together with a brief discussion of relevant precursors. We examine recent projects, from 2010-2015, in greater detail in order to highlight the current state of the art. We follow the survey with a discussion of common themes and methodologies that can be observed in recent work and identify a number of likely directions for future work in this exciting area.

The impact of game content on the player experience is potentially more critical in casual games than in competitive games because of the diminished role of strategic or tactical diversions. Interestingly, until now procedural content generation (PCG) has nevertheless been investigated almost exclusively in the context of competitive, skills-based gaming. This paper therefore opens a new direction for PCG by placing it at the center of an entirely casual flower-breeding game platform called Petalz. That way, the behavior of players and their reactions to different game mechanics in a casual environment driven by PCG can be investigated. In particular, players in Petalz can: 1) trade their discoveries in a global marketplace; 2) respond to an incentive system that awards diversity; and 3) generate real-world 3-D replicas of their evolved flowers. With over 1900 registered online users and 38 646 unique evolved flowers, Petalz showcases the potential for PCG to enable these kinds of casual game mechanics, thus paving the way for continued innovation with PCG in casual gaming.

Traditionally a dedicated level designer has to create every aspect of a level by hand, distribute and gather people to test it. Especially for multiplayer maps, the cyclic process of developing and testing can be quite cumbersome and time consuming. This paper presents a novel approach to provide live generation of levels for such multiplayer maps through procedural content generation and interactive evolution. The approach is demonstrated in the FPSEvolver video game that aims at replicating the look and feel of popular games like Counter-Strike. Without leaving the game a group of players can generate, play and improve levels to fit their particular preferences by voting on a selection of evolving levels. This approach focuses on maintaining the level design principles for modern first-person shooters that encourages good engagements between players, in the popular game mode “bomb defusal”. The presented approach can generate enjoyable maps that adapt to the preferences of players across a range of skill levels. Several distinct types of levels were created during testing, which range from open to closed, and complex to simple, each fitting closely with what the different players consider a good map.

This paper presents the online application Brain-Crafter, in which users can manually build artificial neural networks (ANNs) to control a robot in a maze environment. Users can either start to construct networks from scratch or elaborate on networks created by other users. In particular, BrainCrafter was designed to study how good we as humans are at building ANNs for control problems and if collaborating with other users can facilitate this process. The results in this paper show that (1) some users were in fact able to successfully construct ANNs that solve the navigation tasks, (2) collaboration between users presented difficulties and (3) the human-developed ANNs that managed to solve the task had certain regularities, suggesting that humans can use some of their intuition and spatial understanding in the design of ANNs. Most importantly, the initial results in this paper can serve as a starting point for investigating how to best combine human and machine design capabilities to create more complex artificial brains.

While the success of electronic music often relies on the uniqueness and quality of selected timbres, many musicians struggle with complicated and expensive equipment and techniques to create their desired sounds. Instead, this paper presents a technique for producing novel timbres that are evolved by the musician through interactive evolutionary computation. Each timbre is produced by an oscillator, which is represented by a special type of artificial neural network (ANN) called a compositional pattern producing network (CPPN). While traditional ANNs compute only sigmoid functions at their hidden nodes, CPPNs can theoretically compute any function and can build on those present in traditional synthesizers (e.g. square, sawtooth, triangle, and sine waves functions) to produce completely novel timbres. Evolved with NeuroEvolution of Augmenting Topologies (NEAT), the aim of this paper is to explore the space of potential sounds that can be generated through such compositional sound synthesis networks (CSSNs). To study the effect of evolution on subjective appreciation, participants in a listener study ranked evolved timbres by personal preference, resulting in preferences skewed toward the first and last generations. In the long run, the CSSN's ability to generate a variety of different and rich timbre opens up the intriguing possibility of evolving a complete CSSN-encoded synthesizer.

The co-evolution of morphology and control for virtual creatures enables the creation of a novel form of gameplay and procedural content generation. Starting with a creature evolved to perform a simple task such as locomotion and removing its brain, the remaining body can be employed in a compelling interactive control problem. Just as we enjoy the challenge and reward of mastering helicopter flight or learning to play a musical instrument, learning to control such a creature through manual activation of its actuators presents an engaging and rewarding puzzle. Importantly, the novelty of this challenge is inexhaustible, since the evolution of virtual creatures provides a way to procedurally generate content for such a game. An endless series of creatures can be evolved for a task, then have their brains removed to become the game's next human-control challenge. To demonstrate this new form of gameplay and content generation, a proof-of-concept game--tentatively titled Darwin's Avatars--was implemented using evolved creature content, and user tested. This implementation also provided a unique opportunity to compare human and evolved control of evolved virtual creatures, both qualitatively and quantitatively, with interesting implications for improvements and future work.

An important phenomenon seen in many areas of biological brains and recently in deep learning architectures is a process known as self-organization. For example, in the primary visual cortex, color and orientation maps develop based on lateral inhibitory connectivity patterns and Hebbian learning dynamics. These topographic maps, which are found in all sensory systems, are thought to be a key factor in enabling abstract cognitive representations. This paper shows for the first time that the Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT) method can be seeded to begin evolution with such lateral connectivity, enabling genuine self-organizing dynamics. The proposed approach draws on HyperNEAT's ability to generate a pattern of weights across the connectivity of an artificial neural network (ANN) based on a function of its geometry. Validating this approach, the afferent weights of an ANN self-organize in this paper to form a genuine topographic map of the input space for a simple line orientation task. Most interestingly, this seed can then be evolved further, providing a method to guide the self-organization of weights in a specific way, much as evolution likely guided the self-organizing trajectories of biological brains.

Legged robots can potentially venture beyond the limits of wheeled vehicles. While creating controllers for such robots by hand is possible, evolutionary algorithms are an alternative that can reduce the burden of hand-crafting robotic controllers. Although major evolutionary approaches to legged locomotion can generate oscillations through popular techniques such as continuous time recurrent neural networks (CTRNNs) or sinusoidal input, they typically face a challenge in maintaining long-term stability. The aim of this paper is to address this challenge by introducing an effective alternative based on a new type of neuron called a single-unit pattern generator (SUPG). The SUPG, which is indirectly encoded by a compositional pattern producing network (CPPN) evolved by HyperNEAT, produces a flexible temporal activation pattern that can be reset and repeated at any time through an explicit trigger input, thereby allowing it to dynamically recalibrate over time to maintain stability. The SUPG approach, which is compared to CTRNNs and sinusoidal input, is shown to produce natural-looking gaits that exhibit superior stability over time, thereby providing a new alternative for evolving oscillatory locomotion.

The biological process of ribosomal assembly is one of the most versatile systems in nature. With only a few small building blocks, this natural process is capable of synthesizing the multitude of complex chemicals that form the basis of all organic life. This paper presents a robotics design and manufacturing scheme which seeks to capture some of the versatility of the ribosomal process. In this scheme, a custom "printer" folds a long ribbon of material in which control elements such as motors have been embedded into a morphology that is capable of accomplishing a pre-defined task. The evolved folding patterns are encoded with a special kind of compositional pattern producing network (CPPN), which can compactly describe patterns with regularities such as symmetry, repetition, and repetition with variation. This paper tests the efficacy of this design scheme and the effects of different ribbon lengths on the ability to produce walking robot morphologies. We show that a single strip of material can be folded into a variety of different morphologies displaying different forms of locomotion. Thus, the results presented here suggest a promising new method for the automated design and manufacturing of robotic systems.

The ambulatory capabilities of legged robots offer the potential for access to dangerous and uneven terrain without a risk to human life. However, while machine learning has proven effective at training such robots to walk, a significant limitation of such approaches is that controllers trained for a specific robot are likely to fail when transferred to a robot with a slightly different morphology. This paper confronts this challenge with a novel strategy: Instead of training a controller for a particular quadruped morphology, it evolves a special function (through a method called HyperNEAT) that takes morphology as input and outputs an entire neural network controller fitted to the specific morphology. Once such a relationship is learned the output controllers are able to work on a diversity of different morphologies. Highlighting the unique potential of such an approach, in this paper a neural controller evolved for three different robot morphologies, which differ in the length of their legs, can interpolate to never-seen intermediate morphologies without any further training. Thus this work suggests a new research path towards learning controllers for whole ranges of morphologies: Instead of learning controllers themselves, it is possible to learn the relationship between morphology and control.

Behaviors evolved in simulation are often not robust to variations of their original training environment. Thus often researchers must train explicitly to encourage such robustness. Traditional methods of training for robustness typically apply multiple non-deterministic evaluations with carefully modeled noisy distributions for sensors and effectors. In practice, such training is often computationally expensive and requires crafting accurate models. Taking inspiration from nature, where animals react appropriately to encountered stimuli, this paper introduces a measure called reactivity, i.e. the tendency to seek and react to changes in environmental input, that is applicable in single deterministic trials and can encourage robustness without exposure to noise. The measure is tested in four different maze navigation tasks, where training with reactivity proves more robust than training without noise, and equally or more robust than training with noise when testing with moderate noise levels. In this way, the results demonstrate the counterintuitive fact that sometimes training with no exposure to noise at all can evolve individuals significantly more robust to noise than by explicitly training with noise. The conclusion is that training for reactivity may often be a computationally more efficient means to encouraging robustness in evolved behaviors. © 2012 Massachusetts Institute of Technology.

An ambitious long-term goal for neuroevolution, which studies how artificial evolutionary processes can be driven to produce brain-like structures, is to evolve neurocontrollers with a high density of neurons and connections that can adapt and learn from past experience. Yet while neuroevolution has produced successful results in a variety of domains, the scale of natural brains remains far beyond reach. This paper unifies a set of advanced neuroevolution techniques into a new method called adaptive evolvable-substrate HyperNEAT, which is a step toward more biologically-plausible artificial neural networks (ANNs). The combined approach is able to fully determine the geometry, density, and plasticity of an evolving neuromodulated ANN. These complementary capabilities are demonstrated in a maze-learning task based on similar experiments with animals. The most interesting aspect of this investigation is that the emergent neural structures are beginning to acquire more natural properties, which means that neuroevolution can begin to pose new problems and answer deeper questions about how brains evolved that are ultimately relevant to the field of AI as a whole.

Intelligence in nature is the product of living brains, which are themselves the product of natural evolution. Although researchers in the field of neuroevolution (NE) attempt to recapitulate this process, artificial neural networks (ANNs) so far evolved through NE algorithms do not match the distinctive capabilities of biological brains. The recently introduced hypercube-based neuroevolution of augmenting topologies (HyperNEAT) approach narrowed this gap by demonstrating that the pattern of weights across the connectivity of an ANN can be generated as a function of its geometry, thereby allowing large ANNs to be evolved for high-dimensional problems. Yet the positions and number of the neurons connected through this approach must be decided a priori by the user and, unlike in living brains, cannot change during evolution. Evolvable-substrate HyperNEAT (ES-HyperNEAT), introduced in this article, addresses this limitation by automatically deducing the node geometry from implicit information in the pattern of weights encoded by HyperNEAT, thereby avoiding the need to evolve explicit placement. This approach not only can evolve the location of every neuron in the network, but also can represent regions of varying density, which means resolution can increase holistically over evolution. ES-HyperNEAT is demonstrated through multi-task, maze navigation, and modular retina domains, revealing that the ANNs generated by this new approach assume natural properties such as neural topography and geometric regularity. Also importantly, ES-HyperNEAT's compact indirect encoding can be seeded to begin with a bias toward a desired class of ANN topographies, which facilitates the evolutionary search. The main conclusion is that ES-HyperNEAT significantly expands the scope of neural structures that evolution can discover.

The recently-introduced evolvable-substrate HyperNEAT algorithm (ES-HyperNEAT) demonstrated that the placement and density of hidden nodes in an artificial neural network can be determined based on implicit information in an infinite-resolution pattern of weights, thereby avoiding the need to evolve explicit placement. However, ES-HyperNEAT is computationally expensive because it must search the entire hypercube, and was shown only to match the performance of the original HyperNEAT in a simple benchmark problem. Iterated ES-HyperNEAT, introduced in this paper, helps to reduce computational costs by focusing the search on a sequence of two-dimensional cross-sections of the hypercube and therefore makes possible searching the hypercube at a finer resolution. A series of experiments and an analysis of the evolved networks show for the first time that iterated ES-HyperNEAT not only matches but outperforms original HyperNEAT in more complex domains because ES-HyperNEAT can evolve networks with limited connectivity, elaborate on existing network structure, and compensate for movement of information within the hypercube.

Multirobot domains are a challenge for learning algorithms because they require robots to learn to cooperate to achieve a common goal. The challenge only becomes greater when robots must perform heterogeneous tasks to reach that goal. Multiagent HyperNEAT is a neuroevolutionary method (i.e. a method that evolves neural networks) that has proven successful in several cooperative multiagent domains by exploiting the concept of policy geometry, which means the policies of team members are learned as a function of how they relate to each other based on canonical starting positions. This paper extends the multiagent HyperNEAT algorithm by introducing situational policy geometry, which allows each agent to encode multiple policies that can be switched depending on the agent's state. This concept is demonstrated both in simulation and in real Khepera III robots in a patrol and return task, where robots must cooperate to cover an area and return home when called. Robot teams that are trained with situational policy geometry are compared to teams that are not and shown to find solutions more consistently that are also able to transfer to the real world.

The Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT) approach demonstrated that the pattern of weights across the connectivity of an artificial neural network (ANN) can be generated as a function of its geometry, thereby allowing large ANNs to be evolved for high-dimensional problems. Yet it left to the user the question of where hidden nodes should be placed in a geometry that is potentially infinitely dense. To relieve the user from this decision, this paper introduces an extension called evolvable-substrate HyperNEAT (ES-HyperNEAT) that determines the placement and density of the hidden nodes based on a quadtree-like decomposition of the hypercube of weights and a novel insight about the relationship between connectivity and node placement. The idea is that the representation in HyperNEAT that encodes the pattern of connectivity across the ANN contains implicit information on where the nodes should be placed and can therefore be exploited to avoid the need to evolve explicit placement. In this paper, as a proof of concept, ES-HyperNEAT discovers working placements of hidden nodes for a simple navigation domain on its own, thereby eliminating the need to configure the HyperNEAT substrate by hand and suggesting the potential power of the new approach.

A major goal for researchers in neuroevolution is to evolve artificial neural networks (ANNs) that can learn during their lifetime. Such networks can adapt to changes in their environment that evolution on its own cannot anticipate. However, a profound problem with evolving adaptive systems is that if the impact of learning on the fitness of the agent is only marginal, then evolution is likely to produce individuals that do not exhibit the desired adaptive behavior. Instead, because it is easier at first to improve fitness without evolving the ability to learn, they are likely to exploit domain-dependent static (i.e. non-adaptive) heuristics. This paper proposes a way to escape the deceptive trap of static policies based on the novelty search algorithm, which opens up a new avenue in the evolution of adaptive systems because it can exploit the behavioral difference between learning and non-learning individuals. The main idea in novelty search is to abandon objective-based fitness and instead simply search only for novel behavior, which avoids deception entirely and has shown prior promising results in other domains. This paper shows that novelty search significantly outperforms fitness-based search in a tunably deceptive T-Maze navigation domain because it fosters the emergence of adaptive behavior.