This paper introduces text2midi, an end-to-end model to generate MIDI files from textual descriptions. Leveraging the growing popularity of multimodal generative approaches, text2midi capitalizes on the extensive availability of textual data and the success of large language models (LLMs). Our end-to-end system harnesses the power of LLMs to generate symbolic music in the form of MIDI files. Specifically, we utilize a pretrained LLM encoder to process captions, which then condition an autoregressive transformer decoder to produce MIDI sequences that accurately reflect the provided descriptions. This intuitive and user-friendly method significantly streamlines the music creation process by allowing users to generate music pieces using text prompts. We conduct comprehensive empirical evaluations, incorporating both automated and human studies, that show our model generates MIDI files of high quality that are indeed controllable by text captions that may include music theory terms such as chords, keys, and tempo.

We introduce an extensive new dataset of MIDI files, created by transcribing audio recordings of piano performances into their constituent notes. The data pipeline we use is multi-stage, employing a language model to autonomously crawl and score audio recordings from the internet based on their metadata, followed by a stage of pruning and segmentation using an audio classifier. The resulting dataset contains over one million distinct MIDI files, comprising roughly 100,000 hours of transcribed audio. We provide an in-depth analysis of our techniques, offering statistical insights, and investigate the content by extracting metadata tags, which we also provide. Dataset available at https://github.com/loubbrad/aria-midi.

A common musical composition practice is to develop musical pieces using variations of musical themes. In this study, we present an interactive tool which can generate variations of musical themes in real-time using a variational autoencoder model. Our tool is controllable using semantically meaningful musical attributes via latent space regularisation technique to increase the explainability of the model. The tool is integrated into an industry standard digital audio workstation - Ableton Live - using the Max4Live device framework and can run locally on an average personal CPU rather than requiring a costly GPU cluster. In this way we demonstrate how cutting-edge AI research can be integrated into the exiting workflows of professional and practising musicians for use in the real-world beyond the research lab.

In musical compositions, identifying critical points that reveal atypical and unexpected decisions is valuable from a compositional perspective as these points arguably contribute to the enjoyment of listening to music and are useful for applications such as automatic music generation and music understanding. In this study, we suggest a machine learning-based approach for identifying critical decision points, where we utilise two long short-term memory (LSTM) models that originally function as generative networks and are repurposed in our case to identify critical decision points. These models are trained on musical corpora from the classical period and the 20th century providing different angles to the analysis. We demonstrate this approach using two short musical examples and an excerpt from Chopin's Nocturne in E flat major (Op. 9 No. 2). We compare our suggested machine-learning-based approach to two time series analysis methods as the baselines, evaluate the results, and suggest some future directions for this approach.

In order to advance the field of procedural content generation, and transfer knowledge from academic research to everyday use, we need to develop tools that make generative systems easier to understand and control. In this article, we introduce Danesh, a plugin to the unity game development environment, which helps provide a suite of tools that provide automation or analysis of different aspects of procedural generators. We describe here the features of Danesh, including automatic analysis of generated content, the visualization of generative spaces, automatic parameter discovery, and interface smoothing. We also provide reflections on our development of the tool so far.

We consider multi-solution optimization and generative models for the generation of diverse artifacts and the discovery of novel solutions. In cases where the domain's factors of variation are unknown or too complex to encode manually, generative models can provide a learned latent space to approximate these factors. When used as a search space, however, the range and diversity of possible outputs are limited to the expressivity and generative capabilities of the learned model. We compare the output diversity of a quality diversity evolutionary search performed in two different search spaces: 1) a predefined parameterized space and 2) the latent space of a variational autoencoder model. We find that the search on an explicit parametric encoding creates more diverse artifact sets than searching the latent space. A learned model is better at interpolating between known data points than at extrapolating or expanding towards unseen examples. We recommend using a generative model's latent space primarily to measure similarity between artifacts rather than for search and generation. Whenever a parametric encoding is obtainable, it should be preferred over a learned representation as it produces a higher diversity of solutions.

Most generative systems built in game development are parameter-driven, but the relationship between parameters and the output of the system is often unclear. This makes them frustrating to use for both experts and novices, and as a result generators are often filtered post-hoc, or tweaked through time-consuming trial and error. In this paper we introduce two analytical techniques: smoothness and codependence. We show how these features help analyse the impact of a parameter change on a generative system and suggest ways this could feed back into more intelligent tools that make working with procedural generators more precise and pleasant.

In general, videogame development is a difficult and specialist activity. We believe that providing an immediate feedback cycle (liveness) in the software used to develop games may enable greater productivity, creativity and enjoyment for both professional and amateur creators.There are many different methods for achieving interactivity and immediate feedback in software development, including read-eval-print loops, edit-and-continue debuggers and dataflow programming environments. These approaches have each found success in domains they are well-suited to, but games are particularly challenging due to their interactivity and strict performance requirements. We discuss here the applicability of some of these ideas to game development, and then outline a proposal for a live programming model suited to the unique technical challenges of game development. Our approach seeks to provide an extensible way to automate the process of obtaining feedback, through the use of a reactive programming model and dataflow-style UI. We describe our progress in implementing this design, with reference to a simple example game.

The papers in this special section focus on game design based on artificial intelligence (AI). As researchers working with games, we are often faced with terms that either lack a definition entirely or which are contested and debated as part of their very nature. Many papers have tried and failed to define what “fun” is, from where “creativity” originates, or what “difficulty” means, for example, and many more papers in the future will try and fail to do the same. Tangling with ineffable concepts, with moving targets and nebulous ideas is all part of the joy of doing artificial intelligence (AI) research in a complex, multifaceted domain like games.

Casual creators are a type of design tool identified by Compton & Mateas, characterised by an orientation towards enjoyable, intrinsically motivated creative exploration, rather than task-oriented designer productivity. In our experiments holding rapid game jams with Wevva, a casual creator for mobile game design, we have noticed, however, that users seem to vary considerably even within the context of using a casual creator. Some people focus on designing specific games, while others explore the design space extensively, or even focus exclusively on prodding the edges of the design space looking for its possibilities and limits. We hypothesise that the latter group of users is driven primarily by curiosity about a casual creator and its design space. This results in different patterns of behaviour to the former group (of design-oriented users), which may worth characterising and perhaps explicitly designing for.

Automatically generating content for videogames has long been a staple of game development and the focus of much successful research. Such forays into content generation usually concern themselves with producing a specific game component, such as a level design. This has proven a rich and challenging area of research, but in focusing on creating separate parts of a larger game, we miss out on the most challenging and interesting aspects of game development. By expanding our scope to the automated design of entire games, we can investigate the relationship between the different creative tasks undertaken in game development, tackle the higher level creative challenges of game design, and ultimately build systems capable of much greater novelty, surprise, and quality in their output. This paper, the first in a series of two, describes two case studies in automating game design, proposing cooperative coevolution as a useful technique to use within systems that automate this process. We show how this technique allows essentially separate content generators to produce content that complements each other. We also describe systems that have used this to design games with subtle emergent effects. After introducing the technique and its technical basis in this paper, in the second paper in the series we discuss higher level issues in automated game design, such as potential overlap with computational creativity and the issue of evaluation.

Procedural content generation is generally viewed as a means to an end-a tool employed by designers to overcome technical problems or achieve a particular design goal. When we move from generating single parts of games to automating the entirety of their design, however, we find ourselves facing a far wider and more interesting set of problems than mere generation. When the designer of a game is a piece of software, we face questions about what it means to be a designer, about computational creativity, and about how to assess the growth of these automated game designers and the value of their output. Answering these questions can lead to new ideas in how to generate content procedurally, and produce systems that can further the cutting edge of game design. This paper describes work done to take an automated game designer and advance it towards being a member of a creative community. We outline extensions made to the system to give it more autonomy and creative independence, in order to strengthen claims that the software is acting creatively. We describe and reflect upon the software's participation in the games community, including entering two game development contests, and show the opportunities and difficulties of such engagement. We consider methods for evaluating automated game designers as creative entities, and underline the need for automated game design to be a major frontier in future games research.

Monte Carlo tree search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarize the results from the key game and nongame domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

The xx papers in this special issue focus on the application of computational aesthestics in video games. Recent years have seen a demographic diversification of computer game players as well as the diversiity of player skills. Therefore, the need for tailoring games to individual experiences and aesthetics has become increasingly important.

Computational analysis of player style has significant potential for video game design: it can provide insights into player behavior, as well as the means to dynamically adapt a game to each individual's style of play. To realize this potential, computational methods need to go beyond considerations of challenge and ability and account for aesthetic aspects of player style. We describe here a semiautomatic unsupervised learning approach to modeling player style using multiclass linear discriminant analysis (LDA). We argue that this approach is widely applicable for modeling player style in a wide range of games, including commercial applications, and illustrate it with two case studies: the first for a novel arcade game called Snakeotron, and the second for Rogue Trooper, a modern commercial third-person shooter video game.

We propose to demonstrate the emotionally aware painting fool, a novel system that combines a machine vision system able to recognise emotions with a non-photorealistic rendering (NPR) system to automatically produce portraits of the sitter in an emotionally enhanced style. During the demonstration, the vision system records a short video clip of a person showing a basic emotion. The system then analyses this video clip, locating facial features and tracking their motion. Using this tracking data, the system analyses which emotion was expressed and what the temporal dynamics of the expression were. This information is then passed to the NPR software. The detected emotion is used to choose appropriate (simulated) art materials, colour palettes, abstraction methods and painting styles, so that the rendered image may heighten the emotion being expressed. The live demonstration shows how each element of the emotionally aware painting fool functions and produces a portrait in approximately 7 minutes.