Reproducibility is important for having confidence in evolutionary machine learning algorithms. Although the focus of reproducibility is usually to recreate an aggregate prediction error score using fixed random seeds, this is not sufficient. Firstly, multiple runs of an algorithm, without a fixed random seed, should ideally return statistically equivalent results. Secondly, it should be confirmed whether the expected behaviour of an algorithm matches its actual behaviour, in terms of how an algorithm targets a reduction in prediction error. Confirming the behaviour of an algorithm is not possible when using a total error aggregate score. Using an error decomposition framework as a methodology for improving the reproducibility of results in evolutionary computation addresses both of these factors. By estimating decomposed error using multiple runs of an algorithm and multiple training sets, the framework provides a greater degree of certainty about the prediction error. Also, decomposing error into bias, variance due to the algorithm (internal variance), and variance due to the training data (external variance) more fully characterises evolutionary algorithms. This allows the behaviour of an algorithm to be confirmed. Applying the framework to a number of evolutionary algorithms shows that their expected behaviour can be different to their actual behaviour. Identifying a behaviour mismatch is important in terms of understanding how to further refine an algorithm as well as how to effectively apply an algorithm to a problem.

Genetic programming (GP) is a common method for performing symbolic regression that relies on the use of ephemeral random constants in order to adequately scale predictions. Suitable values for these constants must be drawn from appropriate, but typically unknown, distributions for the problem being modeled. While rarely used with GP,$Z$-score standardization of feature and response spaces often significantly improves the predictive performance of GP by removing scale issues and reducing error due to bias. However, in some cases it is also associated with erratic error due to variance. This article demonstrates that this variance component increases in the presence of gaps at the boundaries of the training data explanatory variable intervals. An initial solution to this problem is proposed that augments training data with pseudo instances located at the boundaries of the intervals. When applied to benchmark problems, particularly with small training samples, this solution reduces error due to variance and, therefore, total error. Augmentation is shown to also stabilize error in larger problems; however, results suggest that standardized GP works well on such problems with little need for training data augmentation.

Symbolic regression is a common application of genetic programming where model structure and corresponding parameters are evolved in unison. In the majority of work exploring symbolic regression, features are used directly without acknowledgement of their relative scale or unit. This paper extends recent work on the importance of standardisation of features when conducting symbolic regression. Specifically, z-score standardisation of input features is applied to both inputs and response to ensure that evolution explores a model space with zero mean and unit variance. This paper demonstrates that standardisation allows a simpler function set to be used without increasing bias. Additionally, it is demonstrated that standardisation can significantly improve the performance of coefficient optimisation through gradient descent to produce accurate models. Through analysis of several benchmark data sets, we demonstrate that feature standardisation enables simple but effective approaches that are comparable in performance to the state-of-the-art in symbolic regression.

Software effort estimation models are typically developed based on an underlying assumption that all data points are equally relevant to the prediction of effort for future projects. The dynamic nature of several aspects of the software engineering process could mean that this assumption does not hold in at least some cases. This study employs three kernel estimator functions to test the stationarity assumption in five software engineering datasets that have been used in the construction of software effort estimation models. The kernel estimators are used in the generation of nonuniform weights which are subsequently employed in weighted linear regression modeling. In each model, older projects are assigned smaller weights while the more recently completed projects are assigned larger weights, to reflect their potentially greater relevance to present or future projects that need to be estimated. Prediction errors are compared to those obtained from uniform models. Our results indicate that, for the datasets that exhibit underlying nonstationary processes, uniform models are more accurate than the nonuniform models; that is, models based on kernel estimator functions are worse than the models where no weighting was applied. In contrast, the accuracies of uniform and nonuniform models for datasets that exhibited stationary processes were essentially equivalent. Our analysis indicates that as the heterogeneity of a dataset increases, the effect of stationarity is overridden. The results of our study also confirm prior findings that the accuracy of effort estimation models is independent of the type of kernel estimator function used in model development.

An error function can be used to select between candidate models but it does not provide a thorough understanding of the behavior of a model. A greater understanding of an algorithm can be obtained by performing a bias-variance decomposition. Splitting the error into bias and variance is effective for understanding a deterministic algorithm such as k-nearest neighbor, which provides the same predictions when performed multiple times using the same data. However, simply splitting the error into bias and variance is not sufficient for nondeterministic algorithms, such as genetic programming (GP), which potentially produces a different model each time it is run, even when using the same data. This article presents an extended bias-variance decomposition that decomposes error into bias, external variance (error attributable to limited sampling of the problem), and internal variance (error due to random actions performed in the algorithm itself). This decomposition is applied to GP to expose the three components of error, providing a unique insight into the role of maximum tree depth, number of generations, size/complexity of function set, and data standardization in influencing predictive performance. The proposed tool can be used to inform targeted improvements for reducing specific components of model error.

This paper presents a novel approach to constructing ensembles for prediction using a bootstrap aggregation (bagging) model. The proposed method uses analogies from ecological modelling to view bootstrap samples as a local adaptation resource in a spatially structured population. Through local competition and breeding, adaptation towards specific bootstrap samples takes place and the resulting ensemble emerges from a single global population in a single run. This makes better use of available computational resources, and negates the need for multiple runs typically required by a bagging approach. We examine the robustness of the method with respect to the number of bootstrap samples in the ensemble, and demonstrate that the resulting method also has a positive effect on bloat control. Finally, the effectiveness of the method relative to existing bagging approaches such as random forests is explored and encouraging performance is demonstrated on a range of benchmark problems.

Difficult benchmark problems are in increasing demand in Genetic Programming (GP). One problem seeing increased usage is the oral bioavailability problem, which is often presented as a challenging problem to both GP and other machine learning methods. However, few properties of the bioavailability data set have been demonstrated, so attributes that make it a challenging problem are largely unknown. This work uncovers important properties of the bioavailability data set, and suggests that the perceived difficulty in this problem can be partially attributed to a lack of pre-processing, including features within the data set that contain no information, and contradictory relationships between the dependent and independent features of the data set. The paper then re-examines the performance of GP on this data set, and contextualises this performance relative to other regression methods. Results suggest that a large component of the observed performance differences on the bioavailability data set can be attributed to variance in the selection of training and testing data. Differences in performance between GP and other methods disappear when multiple training/testing splits are used within experimental work, with performance typically no better than a null modelling approach of reporting the mean of the training data.

Grammatical Evolution (GE) has a long history in evolutionary computation. Central to the behaviour of GE is the use of a linear representation and grammar to map individuals from search spaces into problem spaces. This genotype to phenotype mapping is often argued as a distinguishing property of GE relative to other techniques, such as context-free grammar genetic programming (CFG-GP). Since its initial description, GE research has attempted to incorporate information from the grammar into crossover, mutation, and individual initialisation, blurring the distinction between genotype and phenotype and creating GE variants closer to CFG-GP. This is argued to provide GE with the "best of both worlds", allowing degrees of grammatical bias to be introduced into operators to best suit the given problem. This paper examines the behaviour of three grammar-based search methods on several problems from previous GE research. It is shown that, unlike CFG-GP, the performance of "pure" GE on the examined problems closely resembles that of random search. The results suggest that further work is required to determine the cases where the "best of both worlds" of GE are required over a straight CFG-GP approach.

Software effort estimation (SEE) is a core activity in all software processes and development lifecycles. A range of increasingly complex methods has been considered in the past 30 years for the prediction of effort, often with mixed and contradictory results. The comparative assessment of effort prediction methods has therefore become a common approach when considering how best to predict effort over a range of project types. Unfortunately, these assessments use a variety of sampling methods and error measurements, making comparison with other work difficult. This article proposes an automatically transformed linear model (ATLM) as a suitable baseline model for comparison against SEE methods. ATLM is simple yet performs well over a range of different project types. In addition, ATLM may be used with mixed numeric and categorical data and requires no parameter tuning. It is also deterministic, meaning that results obtained are amenable to replication. These and other arguments for using ATLM as a baseline model are presented, and a reference implementation described and made available. We suggest that ATLM should be used as a baseline of effort prediction quality for all future model comparisons in SEE.

A computational model of a double auction market is introduced and extended to allow a controlled cyclic behaviour in the price signal to be developed. Traders are evolved to maximise profit in this market using Grammatical Evolution, and their properties studied for a range of periods and amplitude of the trend in the price signal. The trader grammar allows decision making based on simple trading rules incorporating the concepts of moving-average oscillators and trading range break-out. The results of this investigation demonstrate that traders evolve a short waiting period between decisions, and that there underlying decision logic reflects the scale of the market price frequency. Evidence is presented that suggests evolving a robust profit-making trader, for a range of price frequency changes, requires the training data to have high frequency variation. More generally, to evolve robust solutions for any complex GP problem, a set of local models or an ensemble and state-based approach, is implied by the results.

A grammatical evolutionary model (GE) is used to evolve trading strategies for a limit-order book model. A modified version of a limit-order book generator, based on the original work of Maslov, is used to produce limit-order book tick data. The evolved trading strategies demonstrate profit-making ability even though the Maslov model is fundamentally based on random behaviour.

During the evolution of solutions using genetic programming (GP) there is generally an increase in average tree size without a corresponding increase in fitness-a phenomenon commonly referred to as bloat. Although previously studied from theoretical and practical viewpoints there has been little progress in deriving controls for bloat which do not explicitly refer to tree size. Here, the use of spatial population structure in combination with local elitist replacement is shown to reduce bloat without a subsequent loss of performance. Theoretical concepts regarding inbreeding and the role of elitism are used to support the described approach. The proposed system behavior is confirmed via extensive computer simulations on benchmark problems. The main practical result is that by placing a population on a torus, with selection defined by a Moore neighborhood and local elitist replacement, bloat can be substantially reduced without compromising performance.

The field of evolutionary art is generally concerned with evolving patterns that have little constraint. This paper describes an evolutionary art system that is constrained to form flag designs, following a set of common design patterns. The resulting genotype representation, genetic operators and forms of user interaction are chosen to allow an exploration of ldquoflag spacerdquo, as well as allowing the user to rapidly focus on aspects of specific designs. The utility of the approach is demonstrated by evolving flag designs using image similarity and by user-directed evolution.

The evolution of a population is determined by many factors, including the geographic separation of individuals in the population (spatial structure), parent selection via assortative mating (biasing who breeds with whom), environmental gradients, founder effects, disturbance, selection, stochastic effects characterised as genetic drift and so on. Ultimately the interest in studying a population of organisms is about characterising parent selection over time. This paper will examine the evolution of a population under the neutral conditions of genetic drift and for a simple selection model. For drift two conditions are considered: the first is for a range of spatial (geographic) constraints defined by a network; the second is through the use of a tagging system that models assortative mate selection. A simple selection model for the OneMax problem is used to illustrate the response of a population to selection pressure. An ancestry network is constructed representing the shared parent interactions over time. This structure is analyzed as a method for characterising the interactions of a population. The approach demonstrates a unified model to characterise population dynamics, independent of the underlying evolutionary constraints.

Spatially-structured evolutionary algorithms (SSEAs) have allowed evolutionary search to be scaled up to increasingly larger and more difficult problems. While their use is becoming more widespread, the basic underlying theory behind them has some omissions. In particular, the effect of random genetic drift on spatial structures is largely unknown. This paper uses models derived from random walk theory to describe the behaviour of a specific class of SSEA. The match between the model and experimental findings is very good. It is the intention that this paper serves as the basis for a more abstract model of drift that encompasses all spatially-structured population models.

This article explores the concept of connectivity and biodiversity by using a simple model of an ecosystem. The model is different from standard artificial life systems since there is no attempt to solve a particular problem, nor is there competition between individuals that drives a coevolution and fitness. Fitness gradually increases throughout the simulation, and mutation rate varies based on population size. Additionally, the system is run with a low mutation rate aiming to produce steady-state behaviour. A simple niche structure at every site allows concepts such as keystone species to be defined, and allows an exploration of diversity and extinction. The article addresses two main issues with this model: does altering the connectivity of neighbourhoods affect the critical point and stability of an ecosystem, and are species-area relationships determined by connectivity?.

The paper describes the initial results of a new form of evolutionary system specifically designed for time series modeling. The system combines a grammatically-based genetic programming system with various optimisation techniques. The system uses the evolutionary system to construct the structure of equations and optimisation techniques to essentially fill in the details. Three forms of optimisation are described: optimisation of constants in an equation; the optimisation of both the constants and variables in an equation; and the use of a hill-climbing mutation to further tune the evolved and optimised equations. Preliminary results indicate that this combination of techniques produces significant improvements in convergence based on the training data, and produces equivalent generalisation on unseen data, for a given number of population member evaluations.

The underlying dynamics of algal species in freshwater systems are a complex non-linear problem. Process-based models have been previously developed to describe the time varying behaviour of chlorophyll-a, a measure of algal concentration, for these systems. This paper describes the application of a genetic programming equation discovery system to study various generalisations of a process-based model based on a time series difference equation.