Ensuring the microbiological safety of large, heterogeneous water distribution systems (WDS) typically requires managing appropriate levels of disinfectant residuals including chlorine. WDS include complex fluid interactions that are nonlinear and noisy, making such maintenance a challenging problem for traditional control algorithms. This paper proposes an evolutionary framework to this problem based on neuroevolution, multi-objective optimization, and surrogate modeling. Neural networks were evolved with NEAT to inject chlorine at strategic locations in the distribution network at select times. NSGA-II was employed to optimize four objectives: minimizing the total amount of chlorine injected, keeping chlorine concentrations homogeneous across the network, ensuring that maximum concentrations did not exceed safe bounds, and distributing the injections regularly over time. Each network was evaluated against a surrogate model, i.e. a neural network trained to emulate EPANET, an industry-level hydraulic WDS simulator that is accurate but infeasible in terms of computational cost to support machine learning. The evolved controllers produced a diverse range of Pareto-optimal policies that could be implemented in practice, outperforming PPO, a standard reinforcement learning method. The results thus suggest a pathway toward improving urban water systems, and highlight the potential of using evolution with surrogate modeling to optimize complex real-world systems.

Evolutionary search is well suited for large-scale biophysical brain modeling, where many parameters with nonlinear interactions and no tractable gradients need to be optimized. Standard evolutionary approaches achieve an excellent fit to MRI data; however, among many possible such solutions, it finds ones that overfit to individual subjects and provide limited predictive power. This paper investigates whether guiding evolution with biological knowledge can help. Focusing on whole-brain Dynamic Mean Field (DMF) models, a baseline where 20 parameters were shared across the brain was compared against a heterogeneous formulation where different sets of 20 parameters were used for the seven canonical brain regions. The heterogeneous model was optimized using four strategies: optimizing all parameters at once, a curricular approach following the hierarchy of brain networks (HICO), a reversed curricular approach, and a randomly shuffled curricular approach. While all heterogeneous strategies fit the data well, only curricular approaches generalized to new subjects. Most importantly, only HICO made it possible to use the parameter sets to predict the subjects' behavioral abilities as well. Thus, by guiding evolution with biological knowledge about the hierarchy of brain regions, HICO demonstrated how domain knowledge can be harnessed to serve the purpose of optimization in real-world domains.

Evolutionary computation can be used to optimize several different aspects of neural network architectures. For instance, the TaylorGLO method discovers novel, customized loss functions, resulting in improved performance, faster training, and improved data utilization. A likely reason is that such functions discourage overfitting, leading to effective regularization. This paper demonstrates theoretically that this is indeed the case for TaylorGLO. Learning rule decomposition reveals that evolved loss functions balance two factors: the pull toward zero error, and a push away from it to avoid overfitting. This is a general principle that may be used to understand other regularization techniques as well (as demonstrated in this paper for label smoothing). The theoretical analysis leads to a constraint that can be utilized to find more effective loss functions in practice; the mechanism also results in networks that are more robust (as demonstrated in this paper with adversarial inputs). The analysis in this paper thus constitutes a first step towards understanding regularization, and demonstrates the power of evolutionary neural architecture search in general.

Most AI systems are black boxes generating reasonable outputs for given inputs. Some domains, however, have explainability and trustworthiness requirements that cannot be directly met by these approaches. Various methods have therefore been developed to interpret black-box models after training. This article advocates an alternative approach where the models are transparent and explainable to begin with. This approach, EVOTER, evolves rule-sets based on extended propositional logic expressions. The approach is evaluated in several prediction/classification and prescription/policy search domains with and without a surrogate. It is shown to discover meaningful rule-sets that perform similarly to black-box models. The rules can provide insight into the domain and make hidden biases explicit. It may also be possible to edit the rules directly to remove biases and add constraints. EVOTER thus forms a promising foundation for building trustworthy AI systems for real-world applications in the future.

Propositional satisfiability (SAT) is an NP-complete problem that impacts many research fields, such as planning, verification, and security. Mainstream modern SAT solvers are based on the Conflict-Driven Clause Learning (CDCL) algorithm. Recent work aimed to enhance CDCL SAT solvers using Graph Neural Networks (GNNs). However, so far this approach either has not made solving more effective, or required substantial GPU resources for frequent online model inferences. Aiming to make GNN improvements practical, this paper proposes an approach called NeuroBack, which builds on two insights: (1) predicting phases (i.e., values) of variables appearing in the majority (or even all) of the satisfying assignments are essential for CDCL SAT solving, and (2) it is sufficient to query the neural model only once for the predictions before the SAT solving starts. Once trained, the offline model inference allows NeuroBack to execute exclusively on the CPU, removing its reliance on GPU resources. To train NeuroBack, a new dataset called DataBack containing 120,286 data samples is created. Finally, NeuroBack is implemented as an enhancement to a state-of-the-art SAT solver called Kissat. As a result, it allowed Kissat to solve 5.2% more problems on the recent SAT competition problem set, SATCOMP-2022. NeuroBack therefore shows how machine learning can be harnessed to improve SAT solving in an effective and practical manner.

Solving societal problems on a global scale requires the collection and processing of ideas and methods from diverse sets of international experts. As the number and diversity of human experts increase, so does the likelihood that elements in this collective knowledge can be combined and refined to discover novel and better solutions. However, it is difficult to identify, combine, and refine complementary information in an increasingly large and diverse knowledge base. This paper argues that artificial intelligence (AI) can play a crucial role in this process. An evolutionary AI framework, termed RHEA, fills this role by distilling knowledge from diverse models created by human experts into equivalent neural networks, which are then recombined and refined in a population-based search. The framework was implemented in a formal synthetic domain, demonstrating that it is transparent and systematic. It was then applied to the results of the XPRIZE Pandemic Response Challenge, in which over 100 teams of experts across 23 countries submitted models based on diverse methodologies to predict COVID-19 cases and suggest non-pharmaceutical intervention policies for 235 nations, states, and regions across the globe. Building upon this expert knowledge, by recombining and refining the 169 resulting policy suggestion models, RHEA discovered a broader and more effective set of policies than either AI or human experts alone, as evaluated based on real-world data. The results thus suggest that AI can play a crucial role in realizing the potential of human expertise in global problem-solving.

With the widespread application of Large Language Models (LLMs) to various domains, concerns regarding the trustworthiness of LLMs in safety-critical scenarios have been raised, due to their unpredictable tendency to hallucinate and generate misinformation. Existing LLMs do not have an inherent functionality to provide the users with an uncertainty/confidence metric for each response it generates, making it difficult to evaluate trustworthiness. Although several studies aim to develop uncertainty quantification methods for LLMs, they have fundamental limitations, such as being restricted to classification tasks, requiring additional training and data, considering only lexical instead of semantic information, and being prompt-wise but not response-wise. A new framework is proposed in this paper to address these issues. Semantic density extracts uncertainty/confidence information for each response from a probability distribution perspective in semantic space. It has no restriction on task types and is "off-the-shelf" for new models and tasks. Experiments on seven state-of-the-art LLMs, including the latest Llama 3 and Mixtral-8x22B models, on four free-form question-answering benchmarks demonstrate the superior performance and robustness of semantic density compared to prior approaches.

A domain-independent problem-solving system based on principles of Artificial Life is introduced. In this system, DIAS, the input and output dimensions of the domain are laid out in a spatial medium. A population of actors, each seeing only part of this medium, solves problems collectively in it. The process is independent of the domain and can be implemented through different kinds of actors. Through a set of experiments on various problem domains, DIAS is shown able to solve problems with different dimensionality and complexity, to require no hyperparameter tuning for new problems, and to exhibit lifelong learning, that is, to adapt rapidly to run-time changes in the problem domain, and to do it better than a standard, noncollective approach. DIAS therefore demonstrates a role for ALife in building scalable, general, and adaptive problem-solving systems.

Neural networks require careful weight initialization to prevent signals from exploding or vanishing. Existing initialization schemes solve this problem in specific cases by assuming that the network has a certain activation function or topology. It is difficult to derive such weight initialization strategies, and modern architectures therefore often use these same initialization schemes even though their assumptions do not hold. This paper introduces AutoInit, a weight initialization algorithm that automatically adapts to different neural network architectures. By analytically tracking the mean and variance of signals as they propagate through the network, AutoInit appropriately scales the weights at each layer to avoid exploding or vanishing signals. Experiments demonstrate that AutoInit improves performance of convolutional, residual, and transformer networks across a range of activation function, dropout, weight decay, learning rate, and normalizer settings, and does so more reliably than data-dependent initialization methods. This flexibility allows AutoInit to initialize models for everything from small tabular tasks to large datasets such as ImageNet. Such generality turns out particularly useful in neural architecture search and in activation function discovery. In these settings, AutoInit initializes each candidate appropriately, making performance evaluations more accurate. AutoInit thus serves as an automatic configuration tool that makes design of new neural network architectures more robust. The AutoInit package provides a wrapper around TensorFlow models and is available at https://github.com/cognizant-ai-labs/autoinit.

In building practical applications of evolutionary computation (EC), two optimizations are essential. First, the parameters of the search method need to be tuned to the domain in order to balance exploration and exploitation effectively. Second, the search method needs to be distributed to take advantage of parallel computing resources. This paper presents BLADE (BLAnket Distributed Evolution) as an approach to achieving both goals simultaneously. BLADE uses blankets (i.e., masks on the genetic representation) to tune the evolutionary operators during the search, and implements the search through hub-and-spoke distribution. In the paper, (1) the blanket method is formalized for the (1 + 1)EA case as a Markov chain process. Its effectiveness is then demonstrated by analyzing dominant and subdominant eigenvalues of stochastic matrices, suggesting a generalizable theory; (2) the fitness-level theory is used to analyze the distribution method; and (3) these insights are verified experimentally on three benchmark problems, showing that both blankets and distribution lead to accelerated evolution. Moreover, a surprising synergy emerges between them: When combined with distribution, the blanket approach achieves more than n-fold speedup with n clients in some cases. The work thus highlights the importance and potential of optimizing evolutionary computation in practical applications.

Population-based search has recently emerged as a possible alternative to Reinforcement Learning (RL) for black-box neural architecture search (NAS). It performs well in practice even though it is not theoretically well understood. In particular, whereas traditional population-based search methods such as evolutionary algorithms (EAs) draw much power from crossover operations, it is difficult to take advantage of them in NAS. The main obstacle is believed to be the permutation problem: The mapping between genotype and phenotype in traditional graph representations is many-to-one, leading to a disruptive effect of standard crossover. This paper presents the first theoretical analysis of the behaviors of mutation, crossover and RL in black-box NAS, and proposes a new crossover operator based on the shortest edit path (SEP) in graph space. The SEP crossover is shown theoretically to overcome the permutation problem, and as a result, have a better expected improvement compared to mutation, standard crossover and RL. Further, it empirically outperform these other methods on state-of-the-art NAS benchmarks. The SEP crossover therefore allows taking full advantage of population-based search in NAS, and the underlying theory can serve as a foundation for deeper understanding of black-box NAS methods in general.

Carefully designed activation functions can improve the performance of neural networks in many machine learning tasks. However, it is difficult for humans to construct optimal activation functions, and current activation function search algorithms are prohibitively expensive. This paper aims to improve the state of the art through three steps: First, the benchmark datasets Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT were created by training convolutional, residual, and vision transformer architectures from scratch with 2,913 systematically generated activation functions. Second, a characterization of the benchmark space was developed, leading to a new surrogate-based method for optimization. More specifically, the spectrum of the Fisher information matrix associated with the model's predictive distribution at initialization and the activation function's output distribution were found to be highly predictive of performance. Third, the surrogate was used to discover improved activation functions in several real-world tasks, with a surprising finding: a sigmoidal design that outperformed all other activation functions was discovered, challenging the status quo of always using rectifier nonlinearities in deep learning. Each of these steps is a contribution in its own right; together they serve as a practical and theoretical foundation for further research on activation function optimization.

As neural network classifiers are deployed in real-world applications, it is crucial that their failures can be detected reliably. One practical solution is to assign confidence scores to each prediction, then use these scores to filter out possible misclassifications. However, existing confidence metrics are not yet sufficiently reliable for this role. This paper presents a new framework that produces a quantitative metric for detecting misclassification errors. This framework, RED, builds an error detector on top of the base classifier and estimates uncertainty of the detection scores using Gaussian Processes. Experimental comparisons with other error detection methods on 125 UCI datasets demonstrate that this approach is effective. Further implementations on two probabilistic base classifiers and two large deep learning architecture in vision tasks further confirm that the method is robust and scalable. Third, an empirical analysis of RED with out-of-distribution and adversarial samples shows that the method can be used not only to detect errors but also to understand where they come from. RED can thereby be used to improve trustworthiness of neural network classifiers more broadly in the future.

Evolutionary algorithms are sensitive to the mutation rate (MR); no single value of this parameter works well across domains. Self-adaptive MR approaches have been proposed but they tend to be brittle: Sometimes they decay the MR to zero, thus halting evolution. To make self-adaptive MR robust, this paper introduces the Group Elite Selection of Mutation Rates (GESMR) algorithm. GESMR co-evolves a population of solutions and a population of MRs, such that each MR is assigned to a group of solutions. The resulting best mutational change in the group, instead of average mutational change, is used for MR selection during evolution, thus avoiding the vanishing MR problem. With the same number of function evaluations and with almost no overhead, GESMR converges faster and to better solutions than previous approaches on a wide range of continuous test optimization problems. GESMR also scales well to high-dimensional neuroevolution for supervised image-classification tasks and for reinforcement learning control tasks. Remarkably, GESMR produces MRs that are optimal in the long-term, as demonstrated through a comprehensive look-ahead grid search. Thus, GESMR and its theoretical and empirical analysis demonstrate how self-adaptation can be harnessed to improve performance in several applications of evolutionary computation.

This paper characterizes the inherent power of evolutionary algorithms. This power depends on the computational properties of the genetic encoding. With some encodings, two parents recombined with a simple crossover operator can sample from an arbitrary distribution of child phenotypes. Such encodings are termed expressive encodings in this paper. Universal function approximators, including popular evolutionary substrates of genetic programming and neural networks, can be used to construct expressive encodings. Remarkably, this approach need not be applied only to domains where the phenotype is a function: Expressivity can be achieved even when optimizing static structures, such as binary vectors. Such simpler settings make it possible to characterize expressive encodings theoretically: Across a variety of test problems, expressive encodings are shown to achieve up to super-exponential convergence speed-ups over the standard direct encoding. The conclusion is that, across evolutionary computation areas as diverse as genetic programming, neuroevolution, genetic algorithms, and theory, expressive encodings can be a key to understanding and realizing the full power of evolution.

In order to deploy autonomous agents in digital interactive environments, they must be able to act robustly in unseen situations. The standard machine learning approach is to include as much variation as possible into training these agents. The agents can then interpolate within their training, but they cannot extrapolate much beyond it. This paper proposes a principled approach where a context module is coevolved with a skill module in the game. The context module recognizes the temporal variation in the game and modulates the outputs of the skill module so that the action decisions can be made robustly even in previously unseen situations. The approach is evaluated in the Flappy Bird and LunarLander video games, as well as in the CARLA autonomous driving simulation. The Context+Skill approach leads to significantly more robust behavior in environments that require extrapolation beyond training. Such a principled generalization ability is essential in deploying autonomous agents in real-world tasks, and can serve as a foundation for continual adaptation as well.

Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. Loss functions are a type of metaknowledge that is crucial to effective training of DNNs, however, their potential role in metalearning has not yet been fully explored. Whereas early work focused on genetic programming (GP) on tree representations, this paper proposes continuous CMA-ES optimization of multivariate Taylor polynomial parameterizations. This approach, TaylorGLO, makes it possible to represent and search useful loss functions more effectively. In MNIST, CIFAR-10, and SVHN benchmark tasks, TaylorGLO finds new loss functions that outperform the standard cross-entropy loss as well as novel loss functions previously discovered through GP, in fewer generations. These functions serve to regularize the learning task by discouraging overfitting to the labels, which is particularly useful in tasks where limited training data is available. The results thus demonstrate that loss function optimization is a productive new avenue for metalearning.

Metalearning of deep neural network (DNN) architectures and hyperparameters has become an increasingly important area of research. At the same time, network regularization has been recognized as a crucial dimension to effective training of DNNs. However, the role of metalearning in establishing effective regularization has not yet been fully explored. There is recent evidence that loss-function optimization could play this role, however it is computationally impractical as an outer loop to full training. This paper presents an algorithm called Evolutionary Population-Based Training (EPBT) that interleaves the training of a DNN's weights with the metalearning of loss functions. They are parameterized using multivariate Taylor expansions that EPBT can directly optimize. Such simultaneous adaptation of weights and loss functions can be deceptive, and therefore EPBT uses a quality-diversity heuristic called Novelty Pulsation as well as knowledge distillation to prevent overfitting during training. On the CIFAR-10 and SVHN image classification benchmarks, EPBT results in faster, more accurate learning. The discovered hyperparameters adapt to the training process and serve to regularize the learning task by discouraging overfitting to the labels. EPBT thus demonstrates a practical instantiation of regularization metalearning based on simultaneous training.

Estimating a person's age from a facial image is a challenging problem with clinical applications. Several medical aesthetics treatments have been developed that alter the skin texture and other facial features, with the goal of potentially improving patient's appearance and perceived age. In this paper, this effect was evaluated using evolutionary neural networks with uncertainty estimation. First, a realistic dataset was obtained from clinical studies that makes it possible to estimate age more reliably than e.g. datasets of celebrity images. Second, a neuroevolution approach was developed that customizes the architecture, learning, and data augmentation hyperparameters and the loss function to this task. Using state-of-the-art computer vision architectures as a starting point, evolution improved their original accuracy significantly, eventually outperforming the best human optimizations in this task. Third, the reliability of the age predictions was estimated using RIO, a Gaussian-Process-based uncertainty model. Evaluation on a real-world Botox treatment dataset shows that the treatment has a quantifiable result: The patients' estimated age is reduced significantly compared to placebo treatments. The study thus shows how AI can be harnessed in a new role: To provide an objective quantitative measure of a subjective perception, in this case the proposed effectiveness of medical aesthetics treatments.

This paper introduces and evaluates a novel training method for neural networks: Dual Variable Learning Rates (DVLR). Building on insights from behavioral psychology, the dual learning rates are used to emphasize correct and incorrect responses differently, thereby making the feedback to the network more specific. Further, the learning rates are varied as a function of the network's performance, thereby making it more efficient. DVLR was implemented on three types of networks: feedforward, convolutional, and residual, and two domains: MNIST and CIFAR-10. The results suggest a consistently improved accuracy, demonstrating that DVLR is a promising, psychologically motivated technique for training neural network models.

Several models have been developed to predict how the COVID-19 pandemic spreads, and how it could be contained with nonpharmaceutical interventions, such as social distancing restrictions and school and business closures. This article demonstrates how evolutionary AI can be used to facilitate the next step, i.e., determining most effective intervention strategies automatically. Through evolutionary surrogate-assisted prescription, it is possible to generate a large number of candidate strategies and evaluate them with predictive models. In principle, strategies can be customized for different countries and locales, and balance the need to contain the pandemic and the need to minimize their economic impact. Early experiments suggest that workplace and school restrictions are the most important and need to be designed carefully. They also demonstrate that results of lifting restrictions can be unreliable, and suggest creative ways in which restrictions can be implemented softly, e.g., by alternating them over time. As more data becomes available, the approach can be increasingly useful in dealing with COVID-19 as well as possible future pandemics.

As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Lossfunction Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.

Multivariate testing has recently emerged as a promising technique in web interface design. In contrast to the standard A/B testing, multivariate approach aims at evaluating a large number of values in a few key variables systematically. The Taguchi method is a practical implementation of this idea, focusing on orthogonal combinations of values. It is the current state of the art in applications such as Adobe Target. This paper evaluates an alternative method: population-based search, i.e. evolutionary optimization. Its performance is compared to that of the Taguchi method in several simulated conditions, including an orthogonal one designed to favor the Taguchi method, and two realistic conditions with dependences between variables. Evolutionary optimization is found to perform significantly better especially in the realistic conditions, suggesting that it forms a good approach for web interface design and other related applications in the future.

Cooperative tasks such as herding and hunting are common among higher animals in nature. A particularly complex example is that of mobbing by spotted hyenas. Through careful coordination, a large number of spotted hyenas can attack a group of lions and successfully steal a kill from them, even though lions are much bigger and stronger. This behavior is more complex than others that hyenas exhibit, and it appears to be heritable. How such behavioral advance can emerge in evolution is a fascinating question; it is difficult to study in nature, but computational simulations can provide insight. In simulation, hyenas initially evolved different levels of boldness, corresponding to simple behaviors such as solo attack, delayed attack, and delayed approach. These behaviors can be seen as stepping stones in constructing the more complex mobbing behavior in later generations. These results suggest a general stepping-stone-based mechanism through which complex coordinated behaviors can arise in humans and animals. This insight should prove useful in building cognitive architectures and team strategies for artificial agents in the future.

Malware detection have used machine learning to detect malware in programs. These applications take in raw or processed binary data to neural network models to classify as benign or malicious files. Even though this approach has proven effective against dynamic changes, such as encrypting, obfuscating and packing techniques, it is vulnerable to specific evasion attacks where that small changes in the input data cause misclassification at test time. This paper proposes a new approach: MDEA, an Adversarial Malware Detection model uses evolutionary optimization to create attack samples to make the network robust against evasion attacks. By retraining the model with the evolved malware samples, its performance improves a significant margin.

The choice of activation function can have a large effect on the performance of a neural network. While there have been some attempts to hand-engineer novel activation functions, the Rectified Linear Unit (ReLU) remains the most commonly-used in practice. This paper shows that evolutionary algorithms can discover novel activation functions that outperform ReLU. A tree-based search space of candidate activation functions is defined and explored with mutation, crossover, and exhaustive search. Experiments on training wide residual networks on the CIFAR-10 and CIFAR-100 image datasets show that this approach is effective. Replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. Optimal performance is achieved when evolution is allowed to customize activation functions to a particular task; however, these novel activation functions are shown to generalize, achieving high performance across tasks. Evolutionary optimization of activation functions is therefore a promising new dimension of metalearning in neural networks.

There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes. This paper introduces a general such approach, called Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for example, a random forest or a neural network trained with gradient descent, and the strategy is a neural network that is evolved to maximize the predictions of the surrogate model. ESP is further extended in this paper to sequential decision-making tasks, which makes it possible to evaluate the framework in reinforcement learning (RL) benchmarks. Because the majority of evaluations are done on the surrogate, ESP is more sample efficient, has lower variance, and lower regret than standard RL approaches. Surprisingly, its solutions are also better because both the surrogate and the strategy network regularize the decision making behavior. ESP thus forms a promising foundation to decision optimization in real-world problems.

Evolution provides a creative fount of complex and subtle adaptations that often surprise the scientists who discover them. However, the creativity of evolution is not limited to the natural world: Artificial organisms evolving in computational environments have also elicited surprise and wonder from the researchers studying them. The process of evolution is an algorithmic process that transcends the substrate in which it occurs. Indeed, many researchers in the field of digital evolution can provide examples of how their evolving algorithms and organisms have creatively subverted their expectations or intentions, exposed unrecognized bugs in their code, produced unexpectedly adaptations, or engaged in behaviors and outcomes, uncannily convergent with ones found in nature. Such stories routinely reveal surprise and creativity by evolution in these digital worlds, but they rarely fit into the standard scientific narrative. Instead they are often treated as mere obstacles to be overcome, rather than results that warrant study in their own right. Bugs are fixed, experiments are refocused, and one-off surprises are collapsed into a single data point. The stories themselves are traded among researchers through oral tradition, but that mode of information transmission is inefficient and prone to error and outright loss. Moreover, the fact that these stories tend to be shared only among practitioners means that many natural scientists do not realize how interesting and lifelike digital organisms are and how natural their evolution can be. To our knowledge, no collection of such anecdotes has been published before. This article is the crowd-sourced product of researchers in the fields of artificial life and evolutionary computation who have provided first-hand accounts of such cases. It thus serves as a written, fact-checked collection of scientifically important and even entertaining stories. In doing so we also present here substantial evidence that the existence and importance of evolutionary surprises extends beyond the natural world, and may indeed be a universal property of all complex evolving systems.

Conversion rate optimization means designing web interfaces such that more visitors perform a desired action (such as register or purchase) on the site. One promising approach, implemented in Sentient Ascend, is to optimize the design using evolutionary algorithms, evaluating each candidate design online with actual visitors. Because such evaluations are costly and noisy, several challenges emerge: How can available visitor traffic be used most efficiently? How can good solutions be identified most reliably? How can a high conversion rate be maintained during optimization? This paper proposes a new technique to address these issues. Traffic is allocated to candidate solutions using a multi-armed bandit algorithm, using more traffic on those evaluations that are most useful. In a best-arm identification mode, the best candidate can be identified reliably at the end of evolution, and in a campaign mode, the overall conversion rate can be optimized throughout the entire evolution process. Multi-armed bandit algorithms thus improve performance and reliability of machine discovery in noisy real-world environments.

Deep neural networks (DNNs) have produced state-of-the-art results in many benchmarks and problem domains. However, the success of DNNs depends on the proper configuration of its architecture and hyperparameters. Such a configuration is difficult and as a result, DNNs are often not used to their full potential. In addition, DNNs in commercial applications often need to satisfy real-world design constraints such as size or number of parameters. To make configuration easier, automatic machine learning (AutoML) systems for deep learning have been developed, focusing mostly on optimization of hyperparameters.

Consumer-grade 3D printers have made the fabrication of aesthetic objects and static assemblies easier, opening the door to automate the design of such objects. However, while static designs are easily produced with 3D printing, functional designs, with moving parts, are more difficult to generate: The search space is high-dimensional, the resolution of the 3D-printed parts is not adequate, and it is difficult to predict the physical behavior of imperfect, 3D-printed mechanisms. An example challenge for automating the design of functional, 3D-printed mechanisms is producing a diverse set of reliable and effective gear mechanisms that could be used after production without extensive post-processing. To meet this challenge, an indirect encoding based on a Recurrent Neural Network (RNN) is proposed and evolved using Novelty Search. The elite solutions of each generation are 3D printed to evaluate their functional performance in a physical test platform. The proposed RNN model successfully discovers sequential design rules that are difficult to discover with other methods. Compared to a direct encoding of gear mechanisms evolved with Genetic Algorithms (GAs), the designs produced by the RNN are geometrically more diverse and functionally more effective, thus forming a promising foundation for the generative design of 3D-printed, functional mechanisms.

Long training times have increasingly become a burden for researchers by slowing down the pace of innovation, with some models taking days or weeks to train. In this paper, a new, general technique is presented that aims to speed up the training process by using a thinned-down training dataset. By leveraging autoencoders and the unique properties of embedding spaces, we are able to filter training datasets to only include only the samples that matter the most. Through evaluation on a standard CIFAR-10 image classification task, this technique is shown to be effective. With this technique, training times can be reduced with a minimal loss in accuracy. Conversely, given a fixed training time budget, the technique was shown to improve accuracy by over 50%. This intelligent dataset sampling technique is a practical tool for achieving better results with large datasets and limited computational budgets.

As deep learning applications continue to become more diverse, an interesting question arises: Can general problem solving arise from jointly learning several such diverse tasks? To approach this question, deep multi-task learning is extended in this paper to the setting where there is no obvious overlap between task architectures. The idea is that any set of (architecture,task) pairs can be decomposed into a set of potentially related subproblems, whose sharing is optimized by an efficient stochastic algorithm. The approach is first validated in a classic synthetic multi-task learning benchmark, and then applied to sharing across disparate architectures for vision, NLP, and genomics tasks. It discovers regularities across these domains, encodes them into sharable modules, and combines these modules systematically to improve performance in the individual tasks. The results confirm that sharing learned functionality across diverse domains and architectures is indeed beneficial, thus establishing a key ingredient for general problem solving in the future.

With increasing traffic in urban areas, it is crucial to examine strategies to reduce traffic network congestion. Popular navigation policies currently tend to select the fastest path available for each vehicle. However, a top-down approach to navigation, which considers the traffic network as a whole, offers several speedup possibilities. Minimizing the average travel time of all vehicles in the network with respect to their separate travel deadlines improves traffic throughput. Because such a strategy does not guarantee an optimal navigation route for individual vehicles, we refer to it as a "selfless" policy and based on this observation we propose the Selfless Traffic Routing (STR) model. Hence, we propose a test bed based on Simulation of Urban MObility (SUMO) that can evaluate the performance of a traffic routing policy based on the average travel time of all vehicle agents in a given traffic grid. Continuously calculating optimal actions for multiple agents in real-time is computationally complex. We therefore introduce a value-based reinforcement learning strategy to achieve the benefits offered by a selfless traffic routing model. We explore how this approach can potentially achieve an optimal balance between action quality and the real-time performance of each decision.

A cyberphysical avatar is a semi-autonomous robot that adjusts to an unstructured environment and performs physical tasks subject to critical timing constraints while under human supervision. This article first realizes a cyberphysical avatar that integrates three key technologies: body-compliant control, neuroevolution, and real-time constraints. Body-compliant control is essential for operator safety, because avatars perform cooperative tasks in close proximity to humans; neuroevolution (NEAT) enables “programming” avatars such that they can be used by non-experts for a large array of tasks, some unforeseen, in an unstructured environment; and real-time constraints are indispensable to provide predictable, bounded-time response in human-avatar interaction. Then, we present a study on the tradeoffs between three design parameters for robotic task systems that must incorporate at least three dimensions: (1) the amount of training effort for robot to perform the task, (2) the time available to complete the task when the command is given, and (3) the quality of the result of the performed task. A tradeoff study in this design space by using the imprecise computation as a framework is to perform a common robotic task, specifically, grasping of unknown objects. The results were validated with a real robot and contribute to the development of a systematic approach for designing robotic task systems that must function in environments like flexible manufacturing systems of the future.

Conversion rate optimization (CRO) means designing an e-commerce web interface so that as many users as possible take a desired action such as registering for an account, requesting a contact, or making a purchase. Such design is usually done by hand, evaluating one change at a time through A/B testing, or evaluating all combinations of two or three variables through multivariate testing. Traditional CRO is thus limited to a small fraction of the design space only. This paper describes Sentient Ascend, an automatic CRO system that uses evolutionary search to discover effective web interfaces given a human-designed search space. Design candidates are evaluated in parallel on line with real users, making it possible to discover and utilize interactions between the design elements that are difficult to identify otherwise. A commercial product since September 2016, Ascend has been applied to numerous web interfaces across industries and search space sizes, with up to four-fold improvements over human design. Ascend can therefore be seen as massively multivariate CRO made possible by AI.

As a classic example of imperfect information games, Heads-Up No-limit Texas Holdem (HUNL) has been studied extensively in recent years. While state-of-the-art approaches based on Nash equilibrium have been successful, they lack the ability to model and exploit opponents effectively. This paper presents an evolutionary approach to discover opponent models based on recurrent neural networks (LSTM) and Pattern Recognition Trees. Experimental results showed that poker agents built in this method can adapt to opponents they have never seen in training and exploit weak strategies far more effectively than Slumbot 2017, one of the cutting-edge Nash-equilibrium-based poker agents. In addition, agents evolved through playing against relatively weak rule-based opponents tied statistically with Slumbot in heads-up matches. Thus, the proposed approach is a promising new direction for building high-performance adaptive agents in HUNL and other imperfect information games.

Multitask learning, i.e. learning several tasks at once with the same neural network, can improve performance in each of the tasks. Designing deep neural network architectures for multitask learning is a challenge: There are many ways to tie the tasks together, and the design choices matter. The size and complexity of this problem exceeds human design ability, making it a compelling domain for evolutionary optimization. Using the existing state of the art soft ordering architecture as the starting point, methods for evolving the modules of this architecture and for evolving the overall topology or routing between modules are evaluated in this paper. A synergetic approach of evolving custom routings with evolved, shared modules for each task is found to be very powerful, significantly improving the state of the art in the Omniglot multitask, multialphabet character recognition domain. This result demonstrates how evolution can be instrumental in advancing deep neural network and complex system design in general.

Consumer-grade printers are widely available, but their ability to print complex objects is limited. Therefore, new designs need to be discovered that serve the same function, but are printable. A representative such problem is to produce a working, reliable mechanical spring. The proposed methodology for discovering solutions to this problem consists of three components: First, an effective search space is learned through a variational autoencoder (VAE); second, a surrogate model for functional designs is built; and third, a genetic algorithm is used to simultaneously update the hyperparameters of the surrogate and to optimize the designs using the updated surrogate. Using a car-launcher mechanism as a test domain, spring designs were 3D-printed and evaluated to update the surrogate model. Two experiments were then performed: First, the initial set of designs for the surrogate-based optimizer was selected randomly from the training set that was used for training the VAE model, which resulted in an exploitative search behavior. On the other hand, in the second experiment, the initial set was composed of more uniformly selected designs from the same training set and a more explorative search behavior was observed. Both of the experiments showed that the methodology generates interesting, successful, and reliable spring geometries robust to the noise inherent in the 3D printing process. The methodology can be generalized to other functional design problems, thus making consumer-grade 3D printing more versatile.

Deep multitask learning boosts performance by sharing learned structure across related tasks. This paper adapts ideas from deep multitask learning to the setting where only a single task is available. The method is formalized as pseudo-task augmentation, in which models are trained with multiple decoders for each task. Pseudo-tasks simulate the effect of training towards closely-related tasks drawn from the same universe. In a suite of experiments, pseudo-task augmentation is shown to improve performance on single-task learning problems. When combined with multitask learning, further improvements are achieved, including state-of-the-art performance on the CelebA dataset, showing that pseudo-task augmentation and multitask learning have complementary value. All in all, pseudo-task augmentation is a broadly applicable and efficient way to boost performance in deep learning systems.

Behavior domination is proposed as a tool for understanding and harnessing the power of evolutionary systems to discover and exploit useful stepping stones. Novelty search has shown promise in overcoming deception by collecting diverse stepping stones, and several algorithms have been proposed that combine novelty with a more traditional fitness measure to refocus search and help novelty search scale to more complex domains. However, combinations of novelty and fitness do not necessarily preserve the stepping stone discovery that novelty search affords. In several existing methods, competition between solutions can lead to an unintended loss of diversity. Behavior domination defines a class of algorithms that avoid this problem, while inheriting theoretical guarantees from multiobjective optimization. Several existing algorithms are shown to be in this class, and a new algorithm is introduced based on fast non-dominated sorting. Experimental results show that this algorithm outperforms existing approaches in domains that contain useful stepping stones, and its advantage is sustained with scale. The conclusion is that behavior domination can help illuminate the complex dynamics of behavior-driven search, and can thus lead to the design of more scalable and robust algorithms.

Conversion optimization means designing a web interface so that as many users as possible take a desired action on it, such as register or purchase. Such design is usually done by hand, testing one change at a time through A/B testing, or a limited number of combinations through multivariate testing, making it possible to evaluate only a small fraction of designs in a vast design space. This paper describes Sentient Ascend, an automatic conversion optimization system that uses evolutionary optimization to create effective web interface designs. Ascend makes it possible to discover and utilize interactions between the design elements that are difficult to identify otherwise. Moreover, evaluation of design candidates is done in parallel online, i.e. with a large number of real users interacting with the system. A case study on an existing media site shows that significant improvements (i.e. over 43%) are possible beyond human design. Ascend can therefore be seen as an approach to massively multivariate conversion optimization, based on a massively parallel interactive evolution.

Capabilities of extrusion-based 3D-printers have progressed significantly, but complex forms are still challenging to print. One major problem is overhanging surfaces. These surfaces require extra support structure to be printed, wasting material and time. Furthermore, delicate parts of the object can be damaged when these structures are removed. One potential solution is to print the object in parts, but decomposition is difficult. This paper proposes an evolutionary approach for determining optimal object decompositions for 3D printing. Two alternative methods, with different complementary strengths, are tested: Multi-objective Genetic Algorithm (MOGA) and Covariance Matrix Adaptation Evolution Strategy (CMA-ES). MOGA is able to evolve a set of decompositions at variable complexity, i.e. number of pieces, whereas CMA-ES is able to find a limited number of comparable decompositions with significantly less computational time.

No Free Lunch (NFL) theorems have been developed in many settings over the last two decades. Whereas NFL is known to be possible in any domain based on set-theoretic concepts, probabilistic versions of NFL are presently believed to be impossible in continuous domains. This article develops a new formalization of probabilistic NFL that is sufficiently expressive to prove the existence of NFL in large search domains, such as continuous spaces or function spaces. This formulation is arguably more complicated than its set-theoretic variants, mostly as a result of the numerous technical complications within probability theory itself. However, a probabilistic conceptualization of NFL is important because stochastic optimization methods inherently need to be evaluated probabilistically. Thus the present study fills an important gap in the study of performance of stochastic optimizers.

A general approach to knowledge transfer is introduced in which an agent controlled by a neural network adapts how it reuses existing networks as it learns in a new domain. Networks trained for a new domain can improve their performance by routing activation selectively through previously learned neural structure, regardless of how or for what it was learned. A neuroevolution implementation of this approach is presented with application to high-dimensional sequential decision-making domains. This approach is more general than previous approaches to neural transfer for reinforcement learning. It is domain-agnostic and requires no prior assumptions about the nature of task relatedness or mappings. The method is analyzed in a stochastic version of the Arcade Learning Environment, demonstrating that it improves performance in some of the more complex Atari 2600 games, and that the success of transfer can be predicted based on a high-level characterization of game dynamics.

Novelty search is a powerful biologically motivated method for discovering successful behaviors especially in deceptive domains, like those in artificial life. This paper extends the biological motivation further by distributing novelty search to run in parallel in multiple islands, with periodic migration among them. In this manner, it is possible to scale novelty search to larger populations and more diverse runs, and also to harness available computing power better. A second extension is to improve novelty searchs ability to solve practical problems by biasing the migration and elitism towards higher fitness. The resulting method, DANS, is shown to find better solutions much faster than pure single-population novelty search, making it a promising candidate for solving deceptive design problems in the real world.

Friction Stir Welding (FSW) is an innovative manufacturing process, which is used to join two pieces of metal with frictional heating and plastic deformation due to stirring action. Melting is avoided during the process, therefore problems related to microstructure phase transformation (i.e., cooling from the liquid phase) are avoided. The temperature distribution in the weld zone, as a function of the heat generation, highly affects the evolution of the residual stresses in the work piece, hence the performance of the final product. Therefore, thermal models play a crucial role in detailed analysis and improvement of this process. In this study, a previously developed and validated three-dimensional steady state thermal model of FS welding of AA2024-T3 plates has been used for evaluating the quality of the candidate solutions. It should be noted that this is a computationally expensive model and closed form formulations (i.e. analytical equations) for the underlying physics are not available, which forces us to use them sparingly during the optimization procedure. A mathematical correlation model, a surrogate in other words, is iteratively constructed to replace the FSW simulations and guide the search towards feasible and promising regions. A new surrogate-based optimization algorithm named EICTS, Expected Improvement with Constrained Tournament Selection has been developed. The striking difference of EICTS from other surrogate based constrained optimization methodologies that it needs to construct only two surrogates, i.e. one for the objective function and another one to handle all constraint functions (i.e., instead of approximating each of them individually). EICTS is first tested on some well-known engineering problems with multiple constraints and finally on the FSW problem briefly mentioned above. Its runtime and convergence performances are compared with EIPF (Expected Improvement with Probability of Feasibility) method and found very promising.

Novelty search and related diversity-driven algorithms provide a promising approach to overcoming deception in complex domains. The behavior characterization (BC) is a critical choice in the application of such algorithms. The BC maps each evaluated individual to a behavior, i.e., some vector representation of what the individual is or does during evaluation. Search is then driven towards diversity in a metric space of these behaviors. BCs are built from hand-designed features that are limited by human expertise, or upon generic descriptors that cannot exploit domain nuance. The main contribution of this paper is an approach that addresses these shortcomings. Generic behaviors are recorded from evolution on several training tasks, and a new BC is learned from them that funnels evolution towards successful behaviors on any further tasks drawn from the domain. This approach is tested in increasingly complex simulated maze-solving domains, where it outperforms both hand-coded and generic BCs, in addition to outperforming objective-based search. The conclusion is that adaptive BCs can improve search in many-task domains with little human expertise.

Reinforcement Learning agents with memory are constructed in this paper by extending neuroevolutionary algorithm NEAT to incorporate LSTM cells, i.e. special memory units with gating logic. Initial evaluation on POMDP tasks indicated that memory solutions obtained by evolving LSTMs outperform traditional RNNs. Scaling neuroevolution of LSTM to deep memory problems is challenging because: (1) the fitness landscape is deceptive, and (2) a large number of associated parameters need to be optimized. To overcome these challenges, a new secondary optimization objective is introduced that maximizes the information (Info-max) stored in the LSTM network. The network training is split into two phases. In the first phase (unsupervised phase), independent memory modules are evolved by optimizing for the info-max objective. In the second phase, the networks are trained by optimizing the task fitness. Results on two different memory tasks indicate that neuroevolution can discover powerful LSTM-based memory solution that outperform traditional RNNs.

In an age-layered evolutionary algorithm, candidates are evaluated on a small number of samples first; if they seem promising, they are evaluated with more samples, up to the entire training set. In this manner, weak candidates can be eliminated quickly, and evolution can proceed faster. In this paper, the fitness-level method is used to derive a theoretical upper bound for the runtime of (k+1) age-layered evolutionary strategy, showing a significant potential speedup compared to a non-layered counterpart. The parameters of the upper bound are estimated experimentally in the 11-Multiplexer problem, verifying that the theory can be useful in configuring age layering for maximum advantage. The predictions are validated in a practical implementation of age layering, confirming that 60-fold speedups are possible with this technique.

Many challenging sequential decision-making problems require agents to master multiple tasks. For instance, game agents may need to gather resources, attack opponents, and defend against attacks. Learning algorithms can thus benefit from having separate policies for these tasks, and from knowing when each one is appropriate. How well this approach works depends on how tightly coupled the tasks are. Three cases are identified: Isolated tasks have distinct semantics and do not interact, interleaved tasks have distinct semantics but do interact, and blended tasks have regions where semantics from multiple tasks overlap. Learning across multiple tasks is studied in this article with Modular Multiobjective NEAT, a neuroevolution framework applied to three variants of the challenging Ms. Pac-Man video game. In the standard blended version of the game, a surprising, highly effective machine-discovered task division surpasses human-specified divisions, achieving the best scores to date in this game. In isolated and interleaved versions of the game, human-specified task divisions are also successful, though the best scores are surprisingly still achieved by machine discovery. Modular neuroevolution is thus shown to be capable of finding useful, unexpected task divisions better than those apparent to a human designer.

The nine papers in this special section focus on the development of physics-based simulation video games (PBSG). The focus is on artificial intelligence for specific PBSGs competitions such as Angry Birds and computational pool, as well as on further developments of physics simulators in order to launch the next generation of PBSGs.

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required: Ms. Pac-Man must escape ghosts when they are threats and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multi-objective NEAT (MM-NEAT) to evolve modular neural networks. Each module defines a separate behavior. The modules are used at different times according to a policy that can be human-designed (i.e. Multitask) or discovered automatically by evolution. The appropriate number of modules can be fixed or discovered using a genetic operator called Module Mutation. Several versions of Module Mutation are evaluated in this paper. Both fixed modular networks and Module Mutation networks outperform monolithic networks and Multitask networks. Interestingly, the best networks dedicate modules to critical behaviors (such as escaping when surrounded after luring ghosts near a power pill) that do not follow the customary division of the game into chasing edible and escaping threat ghosts. The results demonstrate that MM-NEAT can discover interesting and effective behavior for agents in challenging games.

While evolutionary computation is well suited for automatic discovery in engineering, it can also be used to gain insight into how humans and organizations could perform more effectively in competitive problem-solving domains. This paper formalizes human creative problem solving as competitive multi-agent search, and advances the hypothesis that evolutionary computation can be used to discover effective strategies for it. In experiments in a social innovation game (similar to a fantasy sports league), neural networks were first trained to model individual human players. These networks were then used as opponents to evolve better game-play strategies with the NEAT neuroevolution method. Evolved strategies scored significantly higher than the human models by innovating, retaining, and retrieving less and by imitating more, thus providing insight into how performance could be improved in such domains. Evolutionary computation in competitive multi-agent search thus provides a possible framework for understanding and supporting various human creative activities in the future.

A challenge in evolutionary computation is to create representations as evolvable as those in natural evolution. This paper hypothesizes that extinction events, i.e. mass extinctions, can significantly increase evolvability, but only when combined with a divergent search algorithm, i.e. a search driven towards diversity (instead of optimality). Extinctions amplify diversity-generation by creating unpredictable evolutionary bottlenecks. Persisting through multiple such bottlenecks is more likely for lineages that diversify across many niches, resulting in indirect selection pressure for the capacity to evolve. This hypothesis is tested through experiments in two evolutionary robotics domains. The results show that combining extinction events with divergent search increases evolvability, while combining them with convergent search offers no similar benefit. The conclusion is that extinction events may provide a simple and effective mechanism to enhance performance of divergent search algorithms.

Most optimization algorithms must undergo time consuming parameter adaptation in order to optimally solve complex, real-world control tasks. Parameter adaptation is inherently a bilevel optimization problem where the lower level objective function is the performance of the control parameters discovered by an optimization algorithm and the upper level objective function is the performance of the algorithm given its parametrization. In this paper, a novel method called MetaEvolutionary Algorithm (MEA) is presented and shown to be capable of efficiently discovering optimal parameters for neuroevolution to solve control problems. In two challenging examples, double pole balancing and helicopter hovering, MEA discovers optimized parameters that result in better performance than hand tuning and other automatic methods. Bilevel optimization in general and MEA in particular, is thus a promising approach for solving difficult control tasks.

Many challenging sequential decision-making problems require agents to master multiple tasks, such as defense and offense in many games. Learning algorithms thus benefit from having separate policies for these tasks, and from knowing when each one is appropriate. How well the methods work depends on the nature of the tasks: Interleaved tasks are disjoint and have different semantics, whereas blended tasks have regions where semantics from different tasks overlap. While many methods work well in interleaved tasks, blended tasks are difficult for methods with strict, human-specified task divisions, such as Multitask Learning. In such problems, task divisions should be discovered automatically. To demonstrate the power of this approach, the MM-NEAT neuroevolution framework is applied in this paper to two variants of the challenging video game of Ms. Pac-Man. In the simplified interleaved version of the game, the results demonstrate when and why such machine-discovered task divisions are useful. In the standard blended version of the game, a surprising, highly effective machine-discovered task division surpasses human-specified divisions, achieving the best scores to date in this game. Modular neuroevolution is thus a promising technique for discovering multimodal behavior for challenging real-world tasks.

We present a study on the tradeoffs between three design parameters for robotic task systems that function in partially unknown and unstructured environments, and under timing constraints. The design space of these robotic tasks must incorporate at least three dimensions: (1) the amount of training effort to teach the robot to perform the task, (2) the time available to complete the task from the point when the command is given to perform the task, and (3) the quality of the result from performing the task. This paper presents a tradeoff study in this design space for a common robotic task, specifically, grasping of unknown objects in unstructured environments. The imprecise computation model is used to provide a framework for this study. The results were validated with a real robot and contribute to the development of a systematic approach for designing robotic task systems that must function in environments like flexible manufacturing systems of the future.

While neuroevolution has been used successfully to discover effective control policies for intelligent agents, it has been difficult to evolve behavior that is multimodal, i.e. consists of distinctly different behaviors in different situations. This article proposes a new method, Modular NeuroEvolution of Augmenting Topologies (ModNEAT), to meet this challenge. ModNEAT decomposes complex tasks into tractable subtasks and utilizes neuroevolution to learn each subtask. Switch networks are evolved with the subtask networks to arbitrate among them and thus combine separate subtask networks into a complete hierarchical policy. Further, the need for new subtask modules is detected automatically by monitoring fitness of the agent population. Experimental results in the machine learning game of OpenNERO showed that ModNEAT outperforms the non-modular rtNEAT in both agent fitness and training efficiency.

When studying different species in the wild, field biologists can see enormous variation in their behaviors and learning abilities. For example, spotted hyenas and baboons share the same habitat and have similar levels of complexity in their so-cial interactions, but differ widely in how specific vs. general their behaviors are. This paper analyzes two potential factors that lead to this difference: the density of connections in the brain, and the number of generations in prolonged evolution (i.e. after a solution has been found). Using neuroevolution with the NEAT algorithm, network structures with different connectivities were evaluated in recognizing digits and their mirror images. These experiments show that general intel-ligence, i.e. recognition of previously unseen examples, in-creases with increase in connectivity, up to a point. General intelligence also increases with the number of generations in prolonged evolution, even when performance no longer im-proves in the known examples. This outcome suggests that general intelligence depends on specific anatomical and en-vironmental factors. The results from this paper can be used to gain insight into differences in animal behaviors, as well as a guideline for constructing complex general behaviors in artificial agents such as video game bots and physical robots.

Robotic grasping of a target object without advance knowledge of its three-dimensional model is a challenging problem. Many studies indicate that robot learning from demonstration (LfD) is a promising way to improve grasping performance, but complete automation of the grasping task in unforeseen circumstances remains difficult. As an alternative to LfD, this paper leverages limited human supervision to achieve robotic grasping of unknown objects in unforeseen circumstances. The technical question is what form of human supervision best minimizes the effort of the human supervisor. The approach here applies a human-supplied bounding box to focus the robot's visual processing on the target object, thereby lessening the dimensionality of the robot's computer vision processing. After the human supervisor defines the bounding box through the man-machine interface, the rest of the grasping task is automated through a vision-based feature-extraction approach where the dexterous hand learns to grasp objects without relying on pre-computed object models through the NEAT neuroevolution algorithm. Given only low-level sensing data from a commercial depth sensor Kinect, our approach evolves neural networks to identify appropriate hand positions and orientations for grasping novel objects. Further, the machine learning results from simulation have been validated by transferring the training results to a physical robot called Dreamer made by the Meka Robotics company. The results demonstrate that grasping novel objects through exploiting neuroevolution from simulation to reality is possible.

When scaling neuroevolution to complex behaviors, cognitive capabilities such as learning, communication, and memory become increasingly important. However, successfully evolving such cognitive abilities remains difficult. This paper argues that a main cause for such difficulty is deception, i.e. evolution converges to a behavior unrelated to the desired solution. More specifically, cognitive behaviors often require accumulating neural structure that provides no immediate fitness benefit, and evolution often thus converges to non-cognitive solutions. To investigate this hypothesis, a common evolutionary robotics T-Maze domain is adapted in three separate ways to require agents to communicate, remember, and learn. Indicative of deception, evolution driven by objective-based fitness often converges upon simple non-cognitive behaviors. In contrast, evolution driven to explore novel behaviors, i.e. novelty search, often evolves the desired cognitive behaviors. The conclusion is that open-ended methods of evolution may better recognize and reward the stepping stones that are necessary for cognitive behavior to emerge.

Traditional evolved virtual creatures [1] are actuated using unevolved, uniform, invisible drives at joints between rigid segments. In contrast, this paper shows how such conventional actuators can be replaced by evolvable muscle drives that are a part of the creature's physical structure. Such a muscle-drive system replaces control intelligence with meaningful morphological complexity. For instance, the experiments in this paper show that control intelligence sufficient for locomotion or jumping can be moved almost entirely from the brain into the musculature of evolved virtual creatures.

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required to succeed: Ms. Pac-Man must escape ghosts when they are threats, and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multiobjective NEAT to evolve modular neural networks. Each module defines a separate policy; evolution discovers these policies and when to use them. The number of modules can be fixed or learned using a new version of a genetic operator, called Module Mutation, which duplicates an existing module that can then evolve to take on a distinct behavioral identity. Both the fixed modular networks and Module Mutation networks outperform traditional monolithic networks. More interestingly, the best modular networks dedicate modules to critical behaviors that do not follow the customary division of the game into chasing edible and escaping threatening ghosts.

This paper addresses the challenge of learning to play many different video games with little domain-specific knowledge. Specifically, it introduces a neuroevolution approach to general Atari 2600 game playing. Four neuroevolution algorithms were paired with three different state representations and evaluated on a set of 61 Atari games. The neuroevolution agents represent different points along the spectrum of algorithmic sophistication - including weight evolution on topologically fixed neural networks (conventional neuroevolution), covariance matrix adaptation evolution strategy (CMA-ES), neuroevolution of augmenting topologies (NEAT), and indirect network encoding (HyperNEAT). State representations include an object representation of the game screen, the raw pixels of the game screen, and seeded noise (a comparative baseline). Results indicate that direct-encoding methods work best on compact state representations while indirect-encoding methods (i.e., HyperNEAT) allow scaling to higher dimensional representations (i.e., the raw game screen). Previous approaches based on temporal-difference (TD) learning had trouble dealing with the large state spaces and sparse reward gradients often found in Atari games. Neuroevolution ameliorates these problems and evolved policies achieve state-of-the-art results, even surpassing human high scores on three games. These results suggest that neuroevolution is a promising approach to general video game playing (GVGP).

This paper proposes a measure-theoretic framework to study iterative stochastic optimizers that provides theoretical tools to explore how the optimization methods may be improved. Within this framework, optimizers form a closed, convex subset of a normed vector space, implying the existence of a distance metric between any two optimizers and a meaningful and computable spectrum of new optimizers between them. It is shown how the formalism applies to evolutionary algorithms in general. The analytic property of continuity is studied in the context of genetic algorithms, revealing the conditions under which approximations such as meta-modeling or surrogate methods may be effective. These results demonstrate the power of the proposed analytic framework, which can be used to propose and analyze new techniques such as controlled convex combinations of optimizers, meta-optimization of algorithm parameters, and more.

Diversity maintenance techniques in evolutionary computation are designed to mitigate the problem of deceptive local optima by encouraging exploration. However, as problems become more difficult, the heuristic of fitness may become increasingly uninformative. Thus, simply encouraging genotypic diversity may fail to much increase the likelihood of evolving a solution. In such cases, diversity needs to be directed towards potentially useful structures. A representative example of such a search process is novelty search, which builds diversity by rewarding behavioral novelty. In this paper the effectiveness of fitness, novelty, and diversity maintenance objectives are compared in two evolutionary robotics domains. In a biped locomotion domain, genotypic diversity maintenance helps evolve biped control policies that travel farther before falling. However, the best method is to optimize a fitness objective and a behavioral novelty objective together. In the more deceptive maze navigation domain, diversity maintenance is ineffective while a novelty objective still increases performance. The conclusion is that while genotypic diversity maintenance works in well-posed domains, a method more directed by phenotypic information, like novelty search, is necessary for highly deceptive ones.

In the 19 years since Karl Sims' landmark publication on evolving virtual creatures (Sims, 1994), much of the future work he proposed has been implemented, having a significant impact on multiple fields including graphics, evolutionary computation, and artificial life. There has, however been one notable exception to this progress. Despite the potential benefits, there has been no clear increase in the behavioral complexity of evolved virtual creatures (EVCs) beyond the light following demonstrated in Sims' original work.

Neural networks are effective tools to solve prediction, modeling, and control tasks. However, methods to train neural networks have been less successful on control problems that require the network to model intricately structured regions in state space. This paper presents neuroannealing, a method for training neural network controllers on such problems. Neuroannealing is based on evolutionary annealing, a global optimization method that leverages all available information to search for the global optimum. Because neuroannealing retains all intermediate solutions, it is able to represent the fitness landscape more accurately than traditional generational methods and so finds solutions that require greater network complexity. This hypothesis is tested on two problems with fractured state spaces. Such problems are difficult for other methods such as NEAT because they require relatively deep network topology in order to extract the relevant features of the network inputs. Neuroannealing outperforms NEAT on these problems, supporting the hypothesis. Overall, neuroannealing is a promising approach for training neural networks to solve complex practical problems.

This paper describes GRADE, a statistical machine learning system developed to support the work of the graduate admissions committee at the University of Texas at Austin Department of Computer Science (UTCS). In recent years, the number of applications to the UTCS PhD program has become too large to manage with a traditional review process. GRADE uses historical admissions data to predict how likely the committee is to admit each new applicant. It reports each prediction as a score similar to those used by human reviewers, and accompanies each by an explanation of what applicant features most influenced its prediction. GRADE makes the review process more efficient by enabling reviewers to spend most of their time on applicants near the decision boundary and by focusing their attention on parts of each applicant’s file that matter the most. An evaluation over two seasons of PhD admissions indicates that the system leads to dramatic time savings, reducing the total time spent on reviews by at least 74%.

A perceptron-based scaled neural predictor (SNP) was implemented to emphasize the most recent branch histories via the following three approaches: (1) expanding the size of tables that correspond to recent branch histories, (2) scaling the branch histories to increase the weights for the most recent histories but decrease those for the old histories, and (3) expanding most recent branch histories to the whole history path. Furthermore, hash mechanisms, and saturating value for adjusting threshold were tuned to achieve the best prediction accuracy in each case. The resulting extended SNP was tested on well-known floating point and integer benchmarks. Using the SimpleScalar 3.0 simulator, while different features have different impact depending on whether the test is floating point or integer, overall such a well-tuned predictor achieves an improved prediction rate compared to prior approaches.

Learning complex behaviour is a difficult task for any artificial agent. Decomposing a task into multiple sub-tasks, learning the sub-tasks separately, and then learning to use them as a whole is a natural way to reduce the dimensionality and complexity of the task function. This approach is demonstrated on a predator agent in the predator-prey-hunter domain. This extended domain has a new agent, a ‘hunter’, that chases the predators. The evading and chasing behaviours are learnt as separate sub-tasks by separate networks using the NEAT neuro-evolution method. A separate network is then evolved to use these networks based on the situation. Task decomposition using this approach performs significantly better in the predator-prey-hunter domain compared to a monolithic network evolved directly on the whole task.

Communication through vocalizations is used by spotted hyenas and chimpanzees for coordination during hunting and for raising alarm calls in defense (Bullinger et al., 2011; Holekamp et al., 2007). Vocal signals are omni-directional and are therefore more effective than visual communication in these situations. In cooperative tasks, agents use these signals to pro-actively exchange information for common good. A simulated predator-prey domain is considered in this paper- where multiple predator agents exchange real valued messages as an approximation of vocalization in nature. In artificial intelligence, the problem of coordination among multiple predator agents during prey capture is hard because of the non-Markovian environment (Panait and Luke, 2005). Experiments are carried out in this paper to show how information exchange through messaging can make the environment less non-Markovian and improve predator team performance during cooperative hunt. The values of these messages are analyzed to study the emergence of a common communication code among the predator agents. The results in this paper also provide an insight into the constraints under which language evolves in nature.

This paper considers the challenge of enabling agents to learn with as little domain-specific knowledge as possible. The main contribution is HyperNEAT-GGP, a HyperNEAT-based General Game Playing approach to Atari games. By leveraging the geometric regularities present in the Atari game screen, HyperNEAT effectively evolves policies for playing two different Atari games, Asterix and Freeway. Results show that HyperNEAT-GGP outperforms existing benchmarks on these games. HyperNEAT-GGP represents a step towards the ambitious goal of creating an agent capable of learning and seamlessly transitioning between many different tasks.

Social learning is an extension to evolutionary algorithms that enables agents to learn from observations of others in the population. Historically, social learning algorithms have employed a student-teacher model where the behavior of one or more high-fitness agents is used to train a subset of the remaining agents in the population. This paper presents ESL, an egalitarian model of social learning in which agents are not labeled as teachers or students, instead allowing any individual receiving a sufficiently high reward to teach other agents to mimic its recent behavior. We validate our approach through a series of experiments in a robot foraging domain, including comparisons of egalitarian social learning with baseline neuroevolution and a variant of student-teacher social learning. In a complex foraging task, ESL converges to near-optimal strategies faster than either benchmark approach, outperforming both by more than an order of magnitude. The results indicate that egalitarian social learning is a promising new paradigm for social learning in intelligent agents.

Intelligent opponent behavior makes video games interesting to human players. Evolutionary computation can discover such behavior, however, it is challenging to evolve behavior that consists of multiple separate tasks. This paper evaluates three ways of meeting this challenge via neuroevolution: 1) multinetwork learns separate controllers for each task, which are then combined manually; 2) multitask evolves separate output units for each task, but shares information within the network's hidden layer; and 3) mode mutation evolves new output modes, and includes a way to arbitrate between them. Whereas the fist two methods require that the task division be known, mode mutation does not. Results in Front/Back Ramming and Predator/Prey games show that each of these methods has different strengths. Multinetwork is good in both domains, taking advantage of the clear division between tasks. Multitask performs well in Front/Back Ramming, in which the relative difficulty of the tasks is even, but poorly in Predator/Prey, in which it is lopsided. Interestingly, mode mutation adapts to this asymmetry and performs well in Predator/Prey. This result demonstrates how a human-specified task division is not always the best. Altogether the results suggest how human knowledge and learning can be combined most effectively to evolve multimodal behavior.

One promising approach to general-purpose artificial intelligence is neuroevolution, which has worked well on a number of problems from resource optimization to robot control. However, state-of-the-art neuroevolution algorithms like neuroevolution of augmenting topologies (NEAT) have surprising difficulty on problems that are fractured, i.e., where the desired actions change abruptly and frequently. Previous work demonstrated that bias and constraint (e.g., RBF-NEAT and Cascade-NEAT algorithms) can improve learning significantly on such problems. However, experiments in this paper show that relatively unrestricted algorithms (e.g., NEAT) still yield the best performance on problems requiring reactive control. Ideally, a single algorithm would be able to perform well on both fractured and unfractured problems. This paper introduces such an algorithm called SNAP-NEAT that uses adaptive operator selection to integrate strengths of NEAT, RBF-NEAT, and Cascade-NEAT. SNAP-NEAT is evaluated empirically on a set of problems ranging from reactive control to high-level strategy. The results show that SNAP-NEAT can adapt intelligently to the type of problem that it faces, thus laying the groundwork for learning algorithms that can be applied to a wide variety of problems.

Creating agents that behave in complex and believable ways in video games and virtual environments is a difficult task. One solution, shaping, has worked well in evolution of neural networks for agent control in relatively straightforward environments such as the NERO video game, but is very labor intensive. Another solution, coevolution, promises to establish shaping automatically, but it is difficult to control. Although these two approaches have been used separately in the past, they are compatible in principle. This paper shows how shaping can be applied to coevolution to guide it towards more effective behaviors, thus enhancing the power of coevolution in competitive environments. Several automated shaping methods, based on manipulating the fitness function and the game rules, are introduced and tested in a "capture-the-flag"-like environment, where the controller networks for two populations of agents are evolved using the rtNEAT neuroevolution method. Each of these shaping methods as well as their combinations are superior to a control, i.e. direct evolution without shaping. They are effective in different and sometimes incompatible ways, suggesting that different methods may work best in different environments. Using shaping, it should thus be possible to employ coevolution to create intelligent agents for a variety of games.

There is a deep connection between simulated annealing and genetic algorithms with proportional selection. Evolutionary annealing is a novel evolutionary algorithm that makes this connection explicit, resulting in an evolutionary optimization method that can be viewed either as simulated annealing with improved sampling or as a non-Markovian selection mechanism for genetic algorithms with selection over all prior populations. A martingale-based analysis shows that evolutionary annealing is asymptotically convergent and this analysis leads to heuristics for setting learning parameters to optimize the convergence rate. In this work and in parallel work evolutionary annealing is shown to converge faster than other evolutionary algorithms on several benchmark problems, establishing a promising foundation for future theoretical and experimental research into algorithms based on evolutionary annealing.

Many different methods for combining human expertise with machine learning in general, and evolutionary computation in particular, are possible. Which of these methods work best, and do they outperform human design and machine design alone? In order to answer this question, a human-subject experiment for comparing human-assisted machine learning methods was conducted. Three different approaches, i.e. advice, shaping, and demonstration, were employed to assist a powerful machine learning technique (neuroevolution) on a collection of agent training tasks, and contrasted with both a completely manual approach (scripting) and a completely hands-off one (neuroevolution alone). The results show that, (1) human-assisted evolution outperforms a manual scripting approach, (2) unassisted evolution performs consistently well across domains, and (3) different methods of assisting neuroevolution outperform unassisted evolution on different tasks. If done right, human-assisted neuroevolution can therefore be a powerful technique for constructing intelligent agents.

Standard genetic algorithms can discover good fitness regions and later forget them due to their Markovian structure, resulting in suboptimal performance. Real-Space Evolutionary Annealing (REA) hybridizes simulated annealing and genetic algorithms into a provably convergent evolutionary algorithm for Euclidean space that relies on non-Markovian selection. REA selects any previously observed solution from an approximated Boltzmann distribution using a cooling schedule. This method enables REA to escape local optima while retaining information about prior generations. In parallel work, REA has been generalized to arbitrary measure spaces and shown to be asymptotically convergent to the global optima. This paper compares REA experimentally to six popular optimization algorithms, including Differential Evolution, Particle Swarm Optimization, Correlated Matrix Adaptation Evolution Strategies, the real-coded Bayesian Optimization Algorithm, a real-coded genetic algorithm, and simulated annealing. REA converges faster to the global optimum and succeeds more often on two out of three multimodal, non-separable benchmarks and performs strongly on all three. In particular, REA vastly outperforms the real-coded genetic algorithm and simulated annealing, proving that the hybridization is better than either algorithm alone. REA is therefore an interesting and effective algorithm for global optimization of difficult fitness functions.

Symmetry is useful as a constraint in designing complex systems such as distributed controllers for multilegged robots. However, it is often difficult to determine which symmetries are appropriate. It is therefore desirable to design such systems automatically, e.g., by utilizing evolutionary algorithms that produce symmetry through developmental mechanisms. The success of these algorithms depends on how well they explore the space of valid symmetries. This paper presents an approach called evolution of network symmetry and modularity (ENSO) that utilizes group theory to search the space of symmetries effectively. This approach was evaluated by evolving neural network controllers for a quadruped robot in physically realistic simulations. On flat ground, the resulting controllers are as fast as those having hand-designed symmetry, and significantly faster than those without symmetry. On inclined ground, where the appropriate symmetries are difficult to determine manually, ENSO produced significantly faster gaits that also generalize better than those of other approaches. On robots with a more complicated structure including knee joints, ENSO resulted in more regular gaits than the other approaches. These results suggest that ENSO is a promising approach for evolving complex systems with modularity and symmetry.

Different solvers for computationally difficult problems such as satisfiability (SAT) perform best on different instances. Algorithm portfolios exploit this phenomenon by predicting solvers' performance on specific problem instances, then shifting computational resources to the solvers that appear best suited. This paper develops a new approach to the problem of making such performance predictions: natural generative models of solver behavior. Two are proposed, both following from an assumption that problem instances cluster into latent classes: a mixture of multinomial distributions, and a mixture of Dirichlet compound multinomial distributions. The latter model extends the former to capture burstiness, the tendency of solver outcomes to recur. These models are integrated into an algorithm portfolio architecture and used to run standard SAT solvers on competition benchmarks. This approach is found competitive with the most prominent existing portfolio, SATzilla, which relies on domain-specific, hand-selected problem features; the latent class models, in contrast, use minimal domain knowledge. Their success suggests that these models can lead to more powerful and more general algorithm portfolio methods.

In nature there are teams of collaborators and competitors that evolve at the same time, yet computationally they have mostly been studied separately so far. This paper focuses on simultaneous cooperative and competitive coevolution in a complex predator-prey domain. Yong and Miikkulainen’s [3] Multi-Agent ESP architecture is extended to a Multi-Component ESP architecture consisting of multiple cooperating neural networks within an agent. This architecture successfully demonstrates hierarchical cooperation and competition in teams of prey and predators. In sustained coevolution in this complex domain, high-level pursuit-evasion behaviors emerge. In this manner, coevolution of neural networks is shown to scale up to an arms race of multiple competing and cooperating populations, more closely modeling coevolution in nature.

Multiobjective evolutionary algorithms have long been applied to engineering problems. Lately they have also been used to evolve behaviors for intelligent agents. In such applications, it is often necessary to “shape ” the behavior via increasingly difficult tasks. Such shaping requires extensive domain knowledge. An alternative is fitness-based shaping through changing selection pressures, which requires little to no domain knowledge. Two such methods are evaluated in this paper. The first approach, Targeting Unachieved Goals, dynamically chooses when an objective should be used for selection based on how well the population is performing in that objective. The second method, Behavioral Diversity, adds a behavioral diversity objective to the objective set. These approaches are implemented in the popular multiobjective evolutionary algorithm NSGA-II and evaluated in a multiobjective battle domain. Both methods outperform plain NSGA-II in evolution time and final performance, but differ in the profiles of final solution populations. Therefore, both methods should allow multiobjective evolution to be more extensively applied to various agent control problems in the future.

Problems such as the design of distributed controllers are characterized by modularity and symmetry. However, the symmetries useful for solving them are often difficult to determine analytically. This paper presents a nature-inspired approach called Evolution of Network Symmetry and mOdularity (ENSO) to solve such problems. It abstracts properties of generative and developmental systems, and utilizes group theory to represent symmetry and search for it systematically, making it more evolvable than randomly mutating symmetry. This approach is evaluated by evolving controllers for a quadruped robot in physically realistic simulations. On flat ground, the resulting controllers are as effective as those having hand-designed symmetries. However, they are significantly faster when evolved on inclined ground, where the appropriate symmetries are difficult to determine manually. The group-theoretic symmetry mutations of ENSO were also significantly more effective at evolving such controllers than random symmetry mutations. Thus, ENSO is a promising approach for evolving modular and symmetric solutions to distributed control problems, as well as multiagent systems in general.

It has been more than 40 years since the first studies of the secondary visual cortex (V2) were published. However, no concrete hypothesis on how the receptive field of V2 neurons supports general shape processing has been proposed to date. Using a computational model that follows the principle of self-organization, we advance two hypotheses in this letter: (1) typical V2 orientation-selective receptive field contains a primary orientation and a secondary orientation component, forming a corner, a junction, or a cross; and (2) V2 columns with the same primary orientation form contiguous domains, divided into subdomains that prefer different secondary orientations. The first hypothesis is consistent with existing experimental evidence, and both hypotheses can be tested with current techniques in animals. In this manner, computational modeling can be used to provide verifiable predictions that eventually allow us to understand the role of V2 in visual processing.

Evolution of neural networks, or neuroevolution, bas been successful on many low-level control problems such as pole balancing, vehicle control, and collision warning. However, high-level strategy problems that require the integration of multiple sub-behaviors have remained difficult for neuroevolution to solve. This paper proposes the hypothesis that such problems are difficult because they are fractured: the correct action varies discontinuously as the agent moves from state to state. This hypothesis is evaluated on several examples of fractured high-level reinforcement learning domains. Standard neuroevolution methods such as NEAT indeed have difficulty solving them. However, a modification of NEAT that uses radial basis function (RBF) nodes to make precise local mutations to network output is able to do much better. These results provide a better understanding of the different types of reinforcement learning problems and the limitations of current neuroevolution methods. Thus, they lay the groundwork for creating the next generation of neuroevolution algorithms that can learn strategic high-level behavior in fractured domains.

Legged robots are useful in tasks such as search and rescue because they can effectively navigate on rugged terrain. However, it is difficult to design controllers for them that would be stable and robust. Learning the control behavior is difficult because optimal behavior is not known, and the search space is too large for reinforcement learning and for straightforward evolution. As a solution, this paper proposes a modular approach for evolving neural network controllers for such robots. The search space is effectively reduced by exploiting symmetry in the robot morphology, and encoding it into network modules. Experiments involving physically realistic simulations of a quadruped robot produce the same symmetric gaits, such as pronk, pace, bound and trot, that are seen in quadruped animals. Moreover, the robot can transition dynamically to more effective gaits when faced with obstacles. The modular approach also scales well when the number of legs or their degrees of freedom are increased. Evolved non-modular controllers, in contrast, produce gaits resembling crippled animals that are much less effective and do not scale up as a result. Hand-designed controllers are also less effective, especially on an obstacle terrain. These results suggest that the modular approach is effective for designing robust locomotion controllers for multilegged robots.

Kernel-based Bayesian methods for Reinforcement Learning (RL) such as Gaussian Process Temporal Difference (GPTD) are particularly promising because they rigorously treat uncertainty in the value function and make it easy to specify prior knowledge. However, the choice of prior distribution significantly affects the empirical performance of the learning agent, and little work has been done extending existing methods for prior model selection to the online setting. This paper develops Replacing-Kernel RL, an online model selection method for GPTD using sequential Monte-Carlo methods. Replacing-Kernel RL is compared to standard GPTD and tile-coding on several RL domains, and is shown to yield significantly better asymptotic performance for many different kernel families. Furthermore, the resulting kernels capture an intuitively useful notion of prior state covariance that may nevertheless be difficult to capture manually.

Opponent models are necessary in games where the game state is only partially known to the player, since the player must infer the state of the game based on the opponents actions. This paper presents an architecture and a process for developing neural network game players that utilize explicit opponent models in order to improve game play against unseen opponents. The model is constructed as a mixture over a set of cardinal opponents, i.e. opponents that represent maximally distinct game strategies. The model is trained to estimate the likelihood that the opponent will make the same move as each of the cardinal opponents would in a given game situation. Experiments were performed in the game of Guess It, a simple game of imperfect information that has no optimal strategy for defeating specific opponents. Opponent modeling is therefore crucial to play this game well. Both opponent modeling and game-playing neural networks were trained using NeuroEvolution of Augmenting Topologies (NEAT). The results demonstrate that game-playing provided with the model outperform networks not provided with the model when played against the same previously unseen opponents. The cardinal mixture architecture therefore constitutes a promising approach for general and dynamic opponent modeling in game-playing.

Adaptive representations allow evolution to explore the space of phenotypes by choosing the most suitable set of genotypic parameters. Although such an approach is believed to be efficient on complex problems, few empirical studieshave been conducted in such domains. In this paper, three neural network representations, a direct encoding, a complexifying encoding, and an implicit encoding capable of adapting the genotype-phenotype mapping are compared on Nothello, a complex game playing domain from the AAAI General Game Playing Competition. Implicit encoding makes the search more efficient and uses several times fewer parameters. Random mutation leads to highly structured phenotypic variation that is acquired during the course of evolution rather than built into the representation itself. Thus, adaptive representations learn to become evolvable, and furthermore do so in a way that makes search efficient on difficult coevolutionary problems.

A single biological neuron is able to perform complex computations that are highly nonlinear in nature, adaptive, and superior to the perceptron model. A neuron is essentially a nonlinear dynamical system. Its state depends on the interactions among its previous states, its intrinsic properties, and the synaptic input it receives. These factors are included in Hodgkin-Huxley (HH) model, which describes the ionic mechanisms involved in the generation of an action potential. This paper proposes training of an artificial neural network to identify and model the physiological properties of a biological neuron, and mimic its input-output mapping. An HH simulator was implemented to generate the training data. The proposed model was able to mimic and predict the dynamic behavior of the HH simulator under novel stimulation conditions; hence, it can be used to extract the dynamics (in vivo or in vitro) of a neuron without any prior knowledge of its physiology. Such a model can in turn be used as a tool for controlling a neuron in order to study its dynamics for further analysis.

Self-organization of connection patterns within brain areas of animals begins prenatally, and has been shown to depend on internally generated patterns of neural activity. The neural structures continue to develop postnatally through externally driven patterns, when the sensory systems are exposed to stimuli from the environment. The internally generated patterns have been proposed to give the neural system an appropriate bias so that it can learn reliably from complex environmental stimuli. This paper evaluates the hypothesis that complex artificial learning systems can benefit from a similar approach, consisting of initial training with patterns from an evolved pattern generator, followed by training with the actual training set. To test this hypothesis, competitive learning networks were trained for recognizing handwritten digits. The results demonstrate how the approach can improve learning performance by discovering the appropriate initial weight biases, thereby compensating for weaknesses of the learning algorithm. Due to the smaller evolutionary search space, this approach was also found to require much fewer generations than direct evolution of network weights. Since discovering the right biases efficiently is critical for solving large-scale problems with learning, these results suggest that internal training pattern generation is an effective method for constructing complex systems

It is sometimes useful to provide intelligent agents with some degree of stochastic behavior, particularly when used in games and simulators. The less-predictable behavior that results from the randomness can make the agents seem more believable, and would encourage the players or users to address the genuine problems presented by a game or simulator rather than simply learning to exploit the embedded agents' predictability. However, such randomized behavior should not harm performance in the agents' designated tasks. This paper introduces a method, called stochastic sharpening, for training artificial neural networks as stochastic controllers for agents in discrete-state environments. Stochastic sharpening reinforces the representation of confidence values in the outputs of networks with localist encodings, and thus produces networks that recommend alternative actions on the basis of their expected utility. Such networks can be used to introduce stochastic behavior with minimal disruption of task performance, resulting in agents that are more believable and less subject to exploitation based on predictability.

Many serious automobile accidents could be avoided if drivers were warned of impending crashes before they occur. Creating such warning systems by hand, however, is a difficult and time-consuming task. This paper describes three advances toward evolving neural networks with NEAT (NeuroEvolution of Augmenting Topologies) to warn about such crashes in real-world environments. First, NEAT was evaluated in a complex, dynamic simulation with other cars, where it outperformed three hand-coded strawman warning policies and generated warning levels comparable with those of an open-road warning system. Second, warning networks were trained using raw pixel data from a simulated camera. Surprisingly, NEAT was able to generate warning networks that performed similarly to those trained with higher-level input and still outperformed the baseline hand-coded warning policies. Third, the NEAT approach was evaluated in the real world using a robotic vehicle testbed. Despite noisy and ambiguous sensor data, NEAT successfully evolved warning networks using both laser rangefinders and visual sensors. The results in this paper set the stage for developing warning networks for real-world traffic, which may someday save lives in real vehicles.

The Layered Pareto Coevolution Archive (LAPCA) was recently proposed as an effective Coevolutionary Memory (CM) which, under certain assumptions, approximates monotonic progress in coevolution. In this paper, a technique is developed that interfaces the LAPCA algorithm with NeuroEvolution of Augmenting Topologies (NEAT), a method to evolve neural networks with demonstrated efficiency in game playing domains. In addition, the behavior of LAPCA is analyzed for the first time in a complex game-playing domain: evolving neural network controllers for the game Pong. The technique is shown to keep the total number of evaluations in the order of those required by NEAT, making it applicable to complex domains. Pong players evolved with a LAPCA and with the Hall of Fame (HOF) perform equally well, but the LAPCA is shown to require significantly less space than the HOF. Therefore, combining NEAT and LAPCA is found to be an effective approach to coevolution.

Evolutionary algorithms tend to produce solutions that are not evolvable: Although current fitness may be high, further search is impeded as the effects of mutation and crossover become increasingly detrimental. In nature, in addition to having high fitness, organisms have evolvable genomes: phenotypic variation resulting from random mutation is structured and robust. Evolvability is important because it allows the population to produce meaningful variation, leading to efficient search. However, because evolvability does not improve immediate fitness, it must be selected for indirectly. One way to establish such a selection pressure is to change the fitness function systematically. Under such conditions, evolvability emerges only if the representation allows manipulating how genotypic variation maps onto phenotypic variation and if such manipulations lead to detectable changes in fitness. This research forms a framework for understanding how fitness function and representation interact to produce evolvability. Ultimately evolvable encodings may lead to evolutionary algorithms that exhibit the structured complexity and robustness found in nature.

A major challenge in reinforcement learning research is to extend the methods that have worked well on discrete, short-range, low-dimensional problems to continuous, high-diameter, high-dimensional problems, such as robot navigation using high-resolution sensors. Self-organizing distinctive-state abstraction (SODA) is a new, generic method by which a robot in a continuous world can better learn to navigate, by learning a set of high-level features and building temporally extended actions to carry it between distinctive states based on those features. A SODA agent first uses a self-organizing feature map to develop a set of high-level perceptual features while exploring the environment with primitive, local actions. The agent then builds a set of high-level actions composed of generic trajectory-following and hill-climbing control laws that carry it between the states at local maxima of feature activations. In an experiment on a simulated robot navigation task, the SODA agent learns to perform a task requiring 300 small-scale, local actions using as few as nine new, temporally extended actions, significantly improving learning time over navigating with the local actions.

Evolutionary methods are powerful tools in discovering solutions for difficult continuous tasks.When such a solution is encoded over multiple genes, a genetic algorithm faces the difficult credit assignment problem of evaluating how a single gene in a chromosome contributes to the full solution.Typically a single evaluation function is used for the entire chromosome, implicitly giving each gene in the chromosome the same evaluation.This method is inefficient because a gene will get credit for the contribution of all the other genes as well.Accurately measuring the fitness of individual genes in such a large search space requires many trials.This paper instead proposes turning this single complex search problem into a multi-agent search problem, where each agent has the simpler task of discovering a suitable gene.Gene-specific evaluation functions can then be created that have better theoreticaal properties than a single evaluation function over all genes.This method is tested in the difficult double-pole balancing problem, showing that agents using gene-specific evaluation functions can create a successful control policy in 20% fewer trials than the best existing genetic.algorithms.The method is extended to more distributed problems, achieving 95% performance gains over tradition methods in the multi-rover domain.

Wavelet-based image coders like the JPEG2000 standard are the state of the art in image compression. Unlike traditional image coders, however, their performance depends to a large degree on the choice of a good wavelet. Most wavelet-based image coders use standard wavelets that are known to perform well on photographic images. However, these wavelets do not perform as well on other common image classes, like scanned documents or fingerprints. In this paper, a method based on the coevolutionary genetic algorithm introduced in [11] is used to evolve specialized wavelets for fingerprint images. These wavelets are compared to the hand-designed wavelet currently used by the FBI to compress fingerprints. The results show that the evolved wavelets consistently outperform the hand-designed wavelet. Using evolution to adapt wavelets to classes of images can therefore significantly increase the quality of compressed images.

In neuroevolution, a genetic algorithm is used to evolve a neural network to perform a particular task. The standard approach is to evolve a population over a number of generations, and then select the final generation's champion as the end result. However, it is possible that there is valuable information present in the population that is not captured by the champion. The standard approach ignores all such information. One possible solution to this problem is to combine multiple individuals from the final population into an ensemble. This approach has been successful in supervised classification tasks, and in this paper, it is extended to evolutionary reinforcement learning in control problems. The method is evaluated on a challenging extension of the classic pole balancing task, demonstrating that an ensemble can achieve significantly better performance than the champion alone.

The Personal Satellite Assistant (PSA) is a small robot proposed by NASA to assist astronauts who are living and working aboard the space shuttle or space station. To help the astronaut, it has to move around safely. Navigation is made difficult by the arrangement of thrusters. Only forward and leftward thrust is available and rotation will introduce translation. This paper shows how stable navigation can be achieved through neuroevolution in three basic navigation tasks: (1) Stopping autorotation, (2) Turning 90 degrees, and (3) Moving forward to a position. The results show that it is possible to learn to control the PSA stably and efficiently through neuroevolution.

Many serious automobile accidents could be avoided if drivers were warned of impending crashes before they occurred. In this paper, a vehicle warning system is evolved to predict such crashes in the RARS driving simulator. The NeuroEvolution of Augmenting Topologies (NEAT) method is first used to evolve a neural network driver that can autonomously navigate a track without crashing. The network is subsequently impaired, resulting in a driver that occasionally makes mistakes and crashes. Using this impaired driver, a crash predictor is evolved that can predict how far in the future a crash is going to occur, information that can be used to generate an appropriate warning level. The main result is that NEAT can successfully evolve a warning system that takes into account the recent history of inputs and outputs, and therefore makes few errors. Experiments were also run to compare training offline from previously collected data with training online in the simulator. While both methods result in successful warning systems, offline training is both faster and more accurate. Thus, the results in this paper set the stage for developing crash predictors that are both accurate and able to adapt online, which may someday save lives in real vehicles.

Self-organization of brain areas in animals begins prenatally, evidently driven by spontaneously generated internal patterns. The neural structures continue to develop postnatally when the sensory systems are exposed to stimuli from the environment. In this process, prenatal training may give the neural system the appropriate bias so that it can learn reliably under changing environmental stimuli. This paper evaluates the hypothesis that an artificial learning system can benefit from a similar approach, consisting of initial training with patterns from an evolved generator followed by training with the actual training set. Competitive learning networks were trained in recognizing handwritten digits in three ways: through environmental learning only, through evolution only, and through prenatal training with evolved pattern generators followed by environmental learning. The results demonstrate that the evolved pattern generator approach leads to better learning performance, suggesting that complex systems can be constructed effectively in this way.

Feature selection is the process of finding the set of inputs to a machine learning algorithm that will yield the best performance. Developing a way to solve this problem automatically would make current machine learning methods much more useful. Previous efforts to automate feature selection rely on expensive meta-learning or are applicable only when labeled training data is available. This paper presents a novel method called FS-NEAT which extends the NEAT neuroevolution method to automatically determine an appropriate set of inputs for the networks it evolves. By learning the network's inputs, topology, and weights simultaneously, FS-NEAT addresses the feature selection problem without relying on meta-learning or labeled data. Initial experiments in an autonomous car racing simulation demonstrate that FS-NEAT can learn better and faster than regular NEAT. In addition, the networks it evolves are smaller and require fewer inputs. Furthermore, FS-NEAT's performance remains robust even as the feature selection task it faces is made increasingly difficult.

In most modern video games, character behavior is scripted; no matter how many times the player exploits a weakness, that weakness is never repaired. Yet, if game characters could learn through interacting with the player, behavior could improve as the game is played, keeping it interesting. This paper introduces the real-time Neuroevolution of Augmenting Topologies (rtNEAT) method for evolving increasingly complex artificial neural networks in real time, as a game is being played. The rtNEAT method allows agents to change and improve during the game. In fact, rtNEAT makes possible an entirely new genre of video games in which the player trains a team of agents through a series of customized exercises. To demonstrate this concept, the Neuroevolving Robotic Operatives (NERO) game was built based on rtNEAT. In NERO, the player trains a team of virtual robots for combat against other players' teams. This paper describes results from this novel application of machine learning, and demonstrates that rtNEAT makes possible video games like NERO where agents evolve and adapt in real time. In the future, rtNEAT may allow new kinds of educational and training applications through interactive and adapting games.

Two major goals in machine learning are the discovery and improvement of solutions to complex problems. In this paper, we argue that complexification, i.e. the incremental elaboration of solutions through adding new structure, achieves both these goals. We demonstrate the power of complexification through the NeuroEvolution of Augmenting Topologies (NEAT) method, which evolves increasingly complex neural network architectures. NEAT is applied to an open-ended coevolutionary robot duel domain where robot controllers compete head to head. Because the robot duel domain supports a wide range of strategies, and because coevolution benefits from an escalating arms race, it serves as a suitable testbed for studying complexification. When compared to the evolution of networks with fixed structure, complexifying evolution discovers significantly more sophisticated strategies. The results suggest that in order to discover and improve complex solutions, evolution, and search in general, should be allowed to complexify as well as optimize.

We introduce the adaptive team of agents (ATA), a system of homogeneous agents with identical control policies which nevertheless adopt heterogeneous roles appropriate to their environment. ATAs have applications in domains such as games, and can be evolved through neuroevolution. In this paper we show how ATAs can be evolved to solve the problem posed by a simple strategy game and discuss their application to richer environments.

A potentially powerful application of evolutionary computation (EC) is to evolve neural networks for automated control tasks. However, in such tasks environments can be unpredictable and fixed control policies may fail when conditions suddenly change. Thus, there is a need to evolve neural networks that can adapt, i.e. change their control policy dynamically as conditions change. In this paper, we examine two methods for evolving neural networks with dynamic policies. The first method evolves recurrent neural networks with fixed connection weights, relying on internal state changes to lead to changes in behavior. The second method evolves local rules that govern connection weight changes. The surprising experimental result is that the former method can be more effective than evolving networks with dynamic weights, calling into question the intuitive notion that networks with dynamic synapses are necessary for evolving solutions to adaptive tasks.

A major challenge for evolutionary computation is to evolve phenotypes such as neural networks, sensory systems, or motor controllers at the same level of complexity as found in biological organisms. In order to meet this challenge, many researchers are proposing indirect encodings, that is, evolutionary mechanisms where the same genes are used multiple times in the process of building a phenotype. Such gene reuse allows compact representations of very complex phenotypes. Development is a natural choice for implementing indirect encodings, if only because nature itself uses this very process. Motivated by the development of embryos in nature, we define artificial embryogeny (AE) as the subdiscipline of evolutionary computation (EC) in which phenotypes undergo a developmental phase. An increasing number of AE systems are currently being developed, and a need has arisen for a principled approach to comparing and contrasting, and ultimately building, such systems. Thus, in this paper, we develop a principled taxonomy for AE. This taxonomy provides a unified context for long-term research in AE, so that implementation decisions can be compared and contrasted along known dimensions in the design space of embryogenic systems. It also allows predicting how the settings of various AE parameters affect the capacity to efficiently evolve complex phenotypes.

Newborn humans preferentially orient to facelike patterns at birth, but months of experience with faces are required for full face processing abilities to develop. Several models have been proposed for how the interaction of genetic and environmental influences can explain these data. These models generally assume that the brain areas responsible for newborn orienting responses are not capable of learning and are physically separate from those that later learn from real faces. However, it has been difficult to reconcile these models with recent discoveries of face learning in newborns and young infants. We propose a general mechanism by which genetically specified and environment-driven preferences can coexist in the same visual areas. In particular, newborn face orienting may be the result of prenatal exposure of a learning system to internally generated input patterns, such as those found in PGO waves during REM sleep. Simulating this process with the HLISSOM biological model of the visual system, we demonstrate that the combination of learning and internal patterns is an efficient way to specify and develop circuitry for face perception. This prenatal learning can account for the newborn preferences for schematic and photographic images of faces, providing a computational explanation for how genetic influences interact with experience to construct a complex adaptive system.

A novel reinforcement learning algorithm, called symbiotic memetic neuro-evolution (SMNE), is presented for neurocontroller development in nonlinear processes. A highly nonlinear bioreactor process is used in a learning efficiency case study. The use of implicit fitness sharing maintains genetic diversity and induces niching pressure, which enhances the synergetic effect between the global search (symbiotic evolutionary algorithm) and the local search (particle swarm optimisation). SMNE's synergetic effect accelerates learning, which translates to greater economic return for the process industries.

Neuroevolution techniques have been successful in many sequential decision tasks, such as robot control and game playing. This paper aims at establishing whether they can be useful in numerical optimization more generally, by comparing neuroevolution to linear programming in a manufacturing optimization domain. It turns out that neuroevolution can learn to compensate for uncertainty in the data and outperform linear programming when the number of variables in the problem is small and the required precision is low, but the current techniques do not (yet) provide an advantage in problems where many variables must be optimized with high precision.

Neuroevolution, i.e. evolving artificial neural networks with genetic algorithms, has been highly effective in reinforcement learning tasks, particularly those with hidden state information. An important question in neuroevolution is how to gain an advantage from evolving neural network topologies along with weights. We present a method, NeuroEvolution of Augmenting Topologies (NEAT) that outperforms the best fixed-topology methods on a challenging benchmark reinforcement learning task. We claim that the increased efficiency is due to (1) employing a principled method of crossover of different topologies, (2) protecting structural innovation using speciation, and (3) incrementally growing from minimal structure. We test this claim through a series of ablation studies that demonstrate that each component is necessary to the system as a whole and to each other. What results is significantly faster learning. NEAT is also an important contribution to GAs because it shows how it is possible for evolution to both optimize and complexify solutions simultaneously, making it possible to evolve increasingly complex solutions over time, thereby strengthening the analogy with biological evolution.

An important question in neuroevolution is how to gain an advantage from evolving neural network topologies along with weights. We present a method, NeuroEvolution of Augmenting Topologies (NEAT), which outperforms the best fixed-topology method on a challenging benchmark reinforcement learning task. We claim that the increased efficiency is due to (1) employing a principled method of crossover of different topologies, (2) protecting structural innovation using speciation, and (3) incrementally growing from minimal structure. We test this claim through a series of ablation studies that demonstrate that each component is necessary to the system as a whole and to each other. What results is significantly faster learning. NEAT is also an important contribution to GAs because it shows how it is possible for evolution to both optimize and complexify solutions simultaneously, offering the possibility of evolving increasingly complex solutions over generations, and strengthening the analogy with biological evolution.

RF-LISSOM, a self-organizing model of laterally connected orientation maps in the primary visual cortex, was used to study the psychological phenomenon known as the tilt aftereffect. The same self-organizing processes that are responsible for the long-term development of the map are shown to result in tilt aftereffects over short timescales in the adult. The model permits simultaneous observation of large numbers of neurons and connections, making it possible to relate high-level phenomena to low-level events, which is difficult to do experimentally. The results give detailed computational support for the long-standing conjecture that the direct tilt aftereffect arises from adaptive lateral interactions between feature detectors. They also make a new prediction that the indirect effect results from the normalization of synaptic efficacies during this process. The model thus provides a unified computational explanation of self-organization and both the direct and indirect tilt aftereffect in the primary visual cortex.

VISOR is a large connectionist system that shows how visual schemas can be learned, represented and used through mechanisms natural to neural networks. Processing in VISOR is based on cooperation, competition, and parallel bottom-up and top-down activation of schema representations. VISOR is robust against noise and variations in the inputs and parameters. It can indicate the confidence of its analysis, pay attention to important minor differences, and use context to recognize ambiguous objects. Experiments also suggest that the representation and learning are stable, and behavior is consistent with human processes such as priming, perceptual reversal and circular reaction in learning. The schema mechanisms of VISOR can serve as a starting point for building robust high-level vision systems, and perhaps for schema-based motor control and natural language processing systems as well.

This article demonstrates the advantages of a cooperative, coevolutionary search in difficult control problems. The symbiotic adaptive neuroevolution (SANE) system coevolves a population of neurons that cooperate to form a functioning neural network. In this process, neurons assume different but overlapping roles, resulting in a robust encoding of control behavior. SANE is shown to be more efficient and more adaptive and to maintain higher levels of diversity than the more common network-based population approaches. Further empirical studies illustrate the emergent neuron specializations and the different roles the neurons assume in the population.

This article presents a self-organizing neural network model for the simultaneous and cooperative development of topographic receptive fields and lateral interactions in cortical maps. Both afferent and lateral connections adapt by the same Hebbian mechanism in a purely local and unsupervised learning process. Afferent input weights of each neuron self-organize into hill-shaped profiles, receptive fields organize topographically across the network, and unique lateral interaction profiles develop for each neuron. The model demonstrates how patterned lateral connections developed based on correlated activity and explains why lateral connection patterns closely follow receptive field properties such as ocular dominance.

An approach to develop new game-playing strategies based on artificial evolution of neural networks is presented. Evolution was directed to discover strategies in Othello against a random-moving opponent and later against an alpha - beta search program. The networks discovered first a standard positional strategy, and subsequently a mobility strategy, an advanced strategy rarely seen outside of tournaments. The latter discovery demonstrates how evolutionary neural networks can develop novel solutions by turning an initial disadvantage into an advantage in a changed environment.

Symbolic artificial intelligence is motivated by the hypothesis that symbol manipulation is both necessary and sufficient for intelligence. In symbolic systems, knowledge is encoded in terms of explicit symbolic structures, and inferences are based on handcrafted rules that sequentially manipulate these structures. Such systems have been quite successful, for example, in modeling in-depth natural language processing, episodic memory, and symbolic problem solving. However, much of the inferencing for everyday natural language understanding appears to take place immediately, without conscious control, apparently based on associations with past experience. This type of reasoning is difficult to model in the symbolic framework. In contrast, subsymbolic (distributed connectionist) networks represent knowledge in terms of correlations, coded in the weights of the network. For a given input, the network computes the most likely answer given its past experience. A number of human-like information processing properties such as learning from examples, context sensitivity, generalization, robustness of behavior, and intuitive reasoning emerge automatically in subsymbolic systems. The major motivation for subsymbolic AI, therefore, is to give a better account for cognitive phenomena that are statistical, or intuitive, in nature.>

A self-organizing neural network for sequence classification called SARDNET is described and analyzed experimentally. SARDNET extends the Kohonen Feature Map architecture with activation re(cid:173) tention and decay in order to create unique distributed response patterns for different sequences. SARDNET yields extremely dense yet descriptive representations of sequential input in very few train(cid:173) ing iterations. The network has proven successful on mapping ar(cid:173) bitrary sequences of binary and real numbers, as well as phonemic representations of English words. Potential applications include isolated spoken word recognition and cognitive science models of sequence processing.

A neural network model for the self-organization of ocular dominance and lateral connections from binocular input is presented. The self-organizing process results in a network where (1) afferent weights of each neuron or(cid:173) ganize into smooth hill-shaped receptive fields primarily on one of the reti(cid:173) nas, (2) neurons with common eye preference form connected, intertwined patches, and (3) lateral connections primarily link regions of the same eye preference. Similar self-organization of cortical structures has been ob(cid:173) served experimentally in strabismic kittens. The model shows how pat(cid:173) terned lateral connections in the cortex may develop based on correlated activity and explains why lateral connection patterns follow receptive field properties such as ocular dominance.