Nature-inspired methods, such as evolutionary computing and lifetime learning, have shown great promise in advancing autonomous robot design. However, the integration of evolution and learning for the joint optimization of robot morphologies and controllers remains underexplored, particularly in dynamic environments. This paper addresses this gap by investigating the effectiveness of Lamarckian inheritance (a mechanism that allows learned traits to be encoded into the genotype and passed to offspring) in improving robot evolution in nonstationary environments. We compare a Lamarckian system with a traditional Darwinian system, where learned traits are not inherited. Using simulated modular robots in six distinct environmental setups, we analyze the fitness progression, learning ability, and parent-offspring similarity within both systems. Our results demonstrate that the Lamarckian system consistently outperforms the Darwinian system, achieving up to 33% higher fitness in the most challenging conditions. The Lamarckian system also recovers more quickly from environmental changes, showing immediate fitness gains when shifting to complex terrains, whereas the Darwinian system adapts more slowly. Real-world tests validate the robustness of the Lamarckian approach, as the top-performing robots evolved in the most challenging environment exhibit the smallest reality gap. These findings highlight the potential of Lamarckian inheritance as a powerful tool for engineering adaptive robotic systems capable of maintaining high performance in dynamic environments.

Robot evolution systems in which bodies and brains evolve in tandem can be significantly improved by extending them with the ability to learn. Technically, this means that 'newborn' robots are given the opportunity to optimize their inherited brain to control the inherited body adequately. Robots are in an underdeveloped 'infant' stage during this learning stage since their brains and fitness are still being improved. An open issue with regard to this infancy period is that of 'fertility': Should the robot be eligible for mating during the learning stage? This paper explores two distinct approaches from the literature, based on the Triangle of Life (TOL) model, where infant robots cannot produce offspring, and the Morphological Innovation Protection (MIP) mechanism, where they can. The main contribution is a new algorithm, TOL with infant fertility (TOL+IF), inspired by MIP. Experimental comparisons with TOL and MIP show that the new method is superior. TOL+IF is successful not only in producing robots with much higher fitness but also in maintaining the population diversity at higher levels and in evolving different interesting morphologies.

Learning from Demonstrations (LfD) allows robots to learn skills from human users, but its effectiveness can suffer due to sub-optimal teaching, especially from untrained demonstrators. Active LfD aims to improve this by letting robots actively request demonstrations to enhance learning. However, this may lead to frequent context switches between various task situations, increasing the human cognitive load and introducing errors to demonstrations. Moreover, few prior studies in active LfD have examined how these active query strategies may impact human teaching in aspects beyond user experience, which can be crucial for developing algorithms that benefit both robot learning and human teaching. To tackle these challenges, we propose an active LfD method that optimizes the query sequence of online human demonstrations via Curriculum Learning (CL), where demonstrators are guided to provide demonstrations in situations of gradually increasing difficulty. We evaluate our method across four simulated robotic tasks with sparse rewards and conduct a user study$(N=26)$to investigate the influence of active LfD methods on human teaching regarding teaching performance, post-guidance teaching adaptivity, and teaching transferability. Our results show that our method significantly improves learning performance compared to three other LfD baselines in terms of the final success rate of the converged policy and sample efficiency. Additionally, results from our user study indicate that our method significantly reduces the time required from human demonstrators and decreases failed demonstration attempts. It also enhances post-guidance human teaching in both seen and unseen scenarios compared to another active LfD baseline, indicating enhanced teaching performance, greater postguidance teaching adaptivity, and better teaching transferability achieved by our method.

Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.

Reinforcement Learning (RL) has achieved great success in sequential decision-making problems but often requires extensive agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) incorporate external expert demonstrations to aid agent exploration during the learning process. However, these demonstrations, typically collected from human users, are costly and thus often limited in quantity. Therefore, how to select the optimal set of human demonstrations that most effectively aids learning becomes a critical concern. This paper introduces EARLY (Episodic Active Learning from demonstration querY), an algorithm designed to enable a learning agent to generate optimized queries for expert demonstrations in a trajectory-based feature space. EARLY employs a trajectory-level estimate of uncertainty in the agent’s current policy to determine the optimal timing and content for feature-based queries. By querying episodic demonstrations instead of isolated state-action pairs, EARLY enhances the human teaching experience and achieves better learning performance. We validate the effectiveness of our method across three simulated navigation tasks of increasing difficulty. Results indicate that our method achieves expert-level performance in all three tasks, converging over 50% faster than other four baseline methods when demonstrations are generated by simulated oracle policies. A follow-up pilot user study (N = 18) further supports that our method maintains significantly better convergence with human expert demonstrators, while also providing a better user experience in terms of perceived task load and requiring significantly less human time.

Abstract Natural groups of animals, such as swarms of social insects, exhibit astonishing degrees of task specialization, useful for solving complex tasks and for survival. This is supported by phenotypic plasticity: individuals sharing the same genotype that is expressed differently for different classes of individuals, each specializing in one task. In this work, we evolve a swarm of simulated robots with phenotypic plasticity to study the emergence of specialized collective behavior during an emergent perception task. Phenotypic plasticity is realized in the form of heterogeneity of behavior by dividing the genotype into two components, with a different neural network controller associated to each component. The whole genotype, which expresses the behavior of the whole group through the two components, is subject to evolution with a single fitness function. We analyze the obtained behaviors and use the insights provided by these results to design an online regulatory mechanism. Our experiments show four main findings: 1) Heterogeneity improves both robustness and scalability; 2) The sub-groups evolve distinct emergent behaviors. 3) The effectiveness of the whole swarm depends on the interaction between the two sub-groups, leading to a more robust performance than with singular sub-group behavior. 4) The online regulatory mechanism improves overall performance and scalability.

In this paper, we compare Bayesian Optimization, Differential Evolution, and an Evolution Strategy employed as a gait-learning algorithm in modular robots. The motivational scenario is the joint evolution of morphologies and controllers, where "newborn" robots also undergo a learning process to optimize their inherited controllers (without changing their bodies). This context raises the question: How do gait-learning algorithms compare when applied to various morphologies that are not known in advance (and thus need to be treated as without priors)? To answer this question, we use a test suite of twenty different robot morphologies to evaluate our gait-learners and compare their efficiency, efficacy, and sensitivity to morphological differences. The results indicate that Bayesian Optimization and Differential Evolution deliver the same solution quality (walking speed for the robot) with fewer evaluations than the Evolution Strategy. Furthermore, the Evolution Strategy is more sensitive for morphological differences (its efficacy varies more between different morphologies) and is more subject to luck (repeated runs on the same morphology show greater variance in the outcomes).

Jointly optimising both the body and brain of a robot is known to be a challenging task, especially when attempting to evolve designs in simulation that will subsequently be built in the real world. To address this, it is increasingly common to combine evolution with a learning algorithm that can either improve the inherited controllers of new offspring to fine tune them to the new body design or learn them from scratch. In this paper an approach is proposed in which a robot is specified indirectly by two compositional pattern producing networks (CPPN) encoded in a single genome, one which encodes the brain and the other the body. The body part of the genome is evolved using an evolutionary algorithm (EA), with an individual learning algorithm (also an EA) applied to the inherited controller to improve it. The goal of this paper is to determine how to utilise the results of learning process most effectively to improve task performance of the robot. Specifically, three variants are investigated: (1) evolution of the body+controller only; (2) a learning algorithm is applied to the inherited controller with the learned fitness assigned to the genome; (3) learning is applied and the genome is updated with the learned controller, as well as being assigned the learned fitness. Experiments are performed in three different scenarios chosen to favour different bodies and locomotion patterns. It is shown that better performance can be obtained using learning but only if the learned controller is inherited by the offspring.

Learning from Demonstrations (LfD) transfers skills from human teachers to robots. However, data imbalance in demonstrations can bias policies towards majority situations. Previous work attempted to solve this problem after data collection, but few efforts were made to maintain a balanced distribution from the phase of data acquisition. Our method accounts for the influence of robots on human teachers and enables robots to actively guide interaction to approximate demonstration distributions to target distributions. Simulated and real-world experiments validated the method’s efficacy in shaping demonstration distribution into various target distributions and robustness to various levels of uncertainties. Also, our method significantly improved the generalization ability of robot learning when LfD policies were trained with data collected by our method compared to natural data collection.

Human-centered interactive robot tasks (e.g., social greetings and cooperative dressing) are a type of task where humans are involved in task dynamics and performance evaluation. Such tasks require spatial and temporal coordination between agents in real-time, tackling physical limitations from constrained robot bodies, and connecting human user experience with concrete learning objectives to inform algorithm design. To solve these challenges, imitation learning has become a popular approach where by a robot learns to perform a task by imitating how human experts do it (i.e., expert policies). However, previous works tend to isolate the algorithm design from the design of the whole learning pipeline, neglecting its connection with other modules inside the process (like data collection and user-centered subjective evaluation) from the view as a system. Going beyond traditional imitation learning, this work reexamines robot imitation learning in human-centered interactive tasks from the perspective of the whole learning pipeline, ranging from data collection to subjective evaluation. We present a process-oriented framework that consists of a guideline to collect diverse yet representative demonstrations and an interpreter to explain subjective user-centered performance with objective robot-related parameters. We illustrate the steps covered by the framework in a fist-bump greeting task as demonstrative deployment. Results show that our framework is able to identify representative human-centered features to instruct demonstration collection and validate influential robot-centered factors to interpret the gap in subjective performance between the expert policy and the imitator policy.

Abstract This paper investigates human–robot collaboration in a novel setup: a human helps a mobile robot that can move and navigate freely in an environment. Specifically, the human helps by remotely taking over control during the learning of a task. The task is to find and collect several items in a walled arena, and Reinforcement Learning is used to seek a suitable controller. If the human observes undesired robot behavior, they can directly issue commands for the wheels through a game joystick. Experiments in a simulator showed that human assistance improved robot behavior efficacy by 30% and efficiency by 12%. The best policies were also tested in real life, using physical robots. Hardware experiments showed no significant difference concerning the simulations, providing empirical validation of our approach in practice.

Designing controllers for robot swarms is challenging, because human developers have typically no good understanding of the link between the details of a controller that governs individual robots and the swarm behavior that is an indirect result of the interactions between swarm members and the environment. In this paper we investigate whether an evolutionary approach can mitigate this problem. We consider a very challenging task where robots with limited sensing and communication abilities must follow the gradient of an environmental feature and use Differential Evolution to evolve a neural network controller for simulated robots. We conduct a systematic study to measure the flexibility and scalability of the method by varying the size of the arena and number of robots in the swarm. The experiments confirm the feasibility of our approach, the evolved robot controllers induced swarm behavior that solved the task. We found that solutions evolved under the harshest conditions (where the environmental clues were the weakest) were the most flexible and that there is a sweet spot regarding the swarm size. Furthermore, we observed collective motion of the swarm, showcasing truly emergent behavior that was not represented in-and selected for during evolution.

The environment is one of the key factors in the emergence of intelligent creatures, but it has received little attention within the Evolutionary Robotics literature. This article investigates the effects of changing environments on morphological and behavioral traits of evolvable robots. In particular, we extend a previous study by evolving robot populations under diverse changing-environment setups, varying the magnitude, frequency, duration, and dynamics of the changes. The results show that long-lasting effects of early generations occur not only when transitioning from easy to hard conditions, but also when going from hard to easy conditions. Furthermore, we demonstrate how the impact of environmental scaffolding is dependent on the nature of the environmental changes involved.

Evolutionary robotics is concerned with optimizing autonomous robots for one or more specific tasks. Remarkably, the energy needed to operate autonomously is hardly ever considered. This is quite striking because energy consumption is a crucial factor in real-world applications and ignoring this aspect can increase the reality gap. In this paper, we aim to mitigate this problem by extending our robot simulator framework with a model of a battery module and studying its effect on robot evolution. The key idea is to include energy efficiency in the definition of fitness. The robots will need to evolve to achieve high gait speed and low energy consumption. Since our system evolves the robots’ morphologies as well as their controllers, we investigate the effect of the energy extension on the morphologies and on the behavior of the evolved robots. The results show that by including the energy consumption, the evolution is not only able to achieve higher task performance (robot speed), but it reaches good performance faster. Inspecting the evolved robots and their behaviors discloses that these improvements are not only caused by better morphologies, but also by better settings of the robots’ controller parameters.

Abstract Surrogate-based optimization, nature-inspired metaheuristics, and hybrid combinations have become state of the art in algorithm design for solving real-world optimization problems. Still, it is difficult for practitioners to get an overview that explains their advantages in comparison to a large number of available methods in the scope of optimization. Available taxonomies lack the embedding of current approaches in the larger context of this broad field. This article presents a taxonomy of the field, which explores and matches algorithm strategies by extracting similarities and differences in their search strategies. A particular focus lies on algorithms using surrogates, nature-inspired designs, and those created by automatic algorithm generation. The extracted features of algorithms, their main concepts, and search operators, allow us to create a set of classification indicators to distinguish between a small number of classes. The features allow a deeper understanding of components of the search strategies and further indicate the close connections between the different algorithm designs. We present intuitive analogies to explain the basic principles of the search algorithms, particularly useful for novices in this research field. Furthermore, this taxonomy allows recommendations for the applicability of the corresponding algorithms.

The challenge of robotic reproduction – making of new robots by recombining two existing ones – has been recently cracked and physically evolving robot systems have come within reach. Here we address the next big hurdle: producing an adequate brain for a newborn robot. In particular, we address the task of targeted locomotion which is arguably a fundamental skill in any practical implementation. We introduce a controller architecture and a generic learning method to allow a modular robot with an arbitrary shape to learn to walk towards a target and follow this target if it moves. Our approach is validated on three robots, a spider, a gecko, and their offspring, in three real-world scenarios.

This paper studies the effects of different environments on morphological and behavioral properties of evolving populations of modular robots. To assess these properties, a set of morphological and behavioral descriptors was defined and the evolving population mapped in this multi-dimensional space. Surprisingly, the results show that seemingly distinct environments can lead to the same regions of this space, i.e., evolution can produce the same kind of morphologies/behaviors under conditions that humans perceive as quite different. These experiments indicate that demonstrating the 'ground truth' of evolution stating the firm impact of the environment on evolved morphologies is harder in evolutionary robotics than usually assumed.

In the last years, reinforcement learning received a lot of attention. One method to solve reinforcement learning tasks is Neuroevolution, where neural networks are optimized by evolutionary algorithms. A disadvantage of Neuroevolution is that it can require numerous function evaluations, while not fully utilizing the available information from each fitness evaluation. This is especially problematic when fitness evaluations become expensive. To reduce the cost of fitness evaluations, surrogate models can be employed to partially replace the fitness function. The difficulty of surrogate modeling for Neuroevolution is the complex search space and how to compare different networks. To that end, recent studies showed that a kernel based approach, particular with phenotypic distance measures, works well. These kernels compare different networks via their behavior (phenotype) rather than their topology or encoding (genotype). In this work, we discuss the use of surrogate model-based Neuroevolution (SMB-NE) using a phenotypic distance for reinforcement learning. In detail, we investigate a) the potential of SMB-NE with respect to evaluation efficiency and b) how to select adequate input sets for the phenotypic distance measure in a reinforcement learning problem. The results indicate that we are able to considerably increase the evaluation efficiency using dynamic input sets.

Clustering of users underlies many of the personalisation algorithms that are in use nowadays. Such clustering is mostly performed in an offline fashion. For a health and wellbeing setting, offline clustering might however not be suitable, as limited data is often available and patient states can also quickly evolve over time. Existing online clustering algorithms are not suitable for the health domain due to the type of data that involves multiple time series evolving over time. In this paper we propose a new online clustering algorithm called CluStream-GT that is suitable for health applications. By using both artificial and real datasets, we show that the approach is far more efficient compared to regular clustering, with an average speedup of 93%, while only losing 12% in the accuracy of the clustering with artificial data and 3% with real data.

We introduce an end-to-end reinforcement learning (RL) solution for the problem of sending personalized digital health interventions. Previous work has shown that personalized interventions can be obtained through RL using simple, discrete state information such as the recent activity performed. In reality however, such features are often not observed, but instead could be inferred from noisy, low-level sensor information obtained from mobile devices (e.g. accelerometers in mobile phones). One could first transform such raw data into discrete activities, but that could throw away important details and would require training a classifier to infer these discrete activities which would need a labeled training set. Instead, we propose to directly learn intervention strategies for the low-level sensor data end-to-end using deep neural networks and RL. We test our novel approach in a self-developed simulation environment which models, and generates, realistic sensor data for daily human activities and show the short-and long-term efficacy of sending personalized physical workout interventions using RL policies. We compare several different input representations and show that learning using raw sensor data is nearly as effective and much more flexible.

Small robots have numerous interesting applications in domains like industry, education, scientific research, and services. For most applications vision is important, however, the limitations of the computing hardware make this a challenging task. In this paper, we address the problem of real-time object recognition and propose the Fast Regions of Interest Search (FROIS) algorithm to quickly find the ROIs of the objects in small robots with low-performance hardware. Subsequently, we use two methods to analyze the ROIs. First, we develop a Convolutional Neural Network on a desktop and deploy it onto the low-performance hardware for object recognition. Second, we adopt the Histogram of Oriented Gradients descriptor and linear Support Vector Machines classifier and optimize the HOG component for faster speed. The experimental results show that the methods work well on our small robots with Raspberry Pi 3 embedded 1.2 GHz ARM CPUs to recognize the objects. Furthermore, we obtain valuable insights about the trade-offs between speed and accuracy.

Evolutionary robotics using real hardware has been almost exclusively restricted to evolving robot controllers, but the technology for evolvable morphologies is advancing quickly. We discuss a proof-of-concept study to demonstrate real robots that can reproduce. Following a general system plan, we implement a robotic habitat that contains all system components in the simplest possible form. We create an initial population of two robots and run a complete life cycle, resulting in a new robot, parented by the first two. Even though the individual steps are simplified to the maximum, the whole system validates the underlying concepts and provides a generic workflow for the creation of more complex incarnations. This hands-on experience provides insights and helps us elaborate on interesting research directions for future development.

Evolutionary robotics using real hardware is currently restricted to evolving robot controllers, but the technology for evolvable morphologies is advancing quickly. Rapid prototyping (3D printing) and automated assembly are the main enablers of robotic systems where robot offspring can be produced based on a blueprint that specifies the morphologies and the controllers of the parents. This article addresses the problem of gait learning in newborn robots whose morphology is unknown in advance. We investigate a reinforcement learning method and conduct simulation experiments using robot morphologies with different size and complexity. We establish that reinforcement learning does the job well and that it outperforms two alternative algorithms. The experiments also give insights into the online dynamics of gait learning and into the influence of the size, shape, and morphological complexity of the modular robots. These insights can potentially be used to predict the viability of modular robotic organisms before they are constructed.

Adapting the control systems of robots on the fly is important in robotic systems of the future. In this paper we present and investigate a three-fold adaptive system based on evolution, individual and social learning in a group of robots and report on a proof-of-concept study based on e-pucks. We distinguish inheritable and learnable components in the robots' makeup, specify and implement operators for evolution, learning and social learning, and test the system in an arena where the task is to learn to avoid obstacles. In particular, we make the sensory layout evolvable, the locomotion control system learnable and investigate the effects of including social learning in the `adaptation engine'. Our simulation experiments demonstrate that the full mix of three adaptive mechanisms is practicable and that adding social learning leads to better controllers faster.

More than a decade after the first extensive overview on parameter control, we revisit the field and present a survey of the state-of-the-art. We briefly summarize the development of the field and discuss existing work related to each major parameter or component of an evolutionary algorithm. Based on this overview, we observe trends in the area, identify some (methodological) shortcomings, and give recommendations for future research.

Parameter control in Evolutionary Computing stands for an approach to parameter setting that changes the parameters of an Evolutionary Algorithm (EA) on-the-fly during the run. In this paper we address the issue of a generic and parameter-independent controller that can be readily plugged into an existing EA and offer performance improvements by varying the EA parameters during the problem solution process. Our approach is based on a careful study of Reinforcement Learning (RL) theory and the use of existing RL techniques. We present experiments using various state-of-the-art EAs solving different difficult problems. Results show that our RL control method has very good potential in improving the quality of the solution found without requiring additional resources or time and with minimal effort from the designer of the application.

Maintaining population diversity is an important and difficult task in Evolutionary Computation in general and Evolutionary Art in particular. A lack of population diversity will result in inefficient search behaviour and premature convergence. In this paper we investigate the effect of using spatially structured populations on population diversity in Evolutionary Art. To this end, we perform several experiments with unsupervised evolution (no human in the loop) of aesthetically pleasing images using a panmictic model Evolutionary Algorithm, a distributed Island Model (with a Best-First selection scheme and with the Multikulti algorithm) and a Cellular Evolutionary Algorithm. In our Island Models experiments we use a number of different parameters settings for number of islands, island size, migration interval, migration size, and initialisation methods. In our Cellular EA experiments we use different settings for width, height and neighbourhood. We also compare the use of structured populations with the use of a panmictic EA with enhanced genetic operators. We find that the use of structured populations is beneficial for maintaining both phenotype and genotype diversity. All configurations of Island Models and Cellular EA outperform our standard panmictic EA on population diversity.

Parameter control mechanisms in evolutionary algorithms (EAs) dynamically change the values of the EA parameters during a run. Research over the last two decades has delivered ample examples where an EA using a parameter control mechanism outperforms its static version with fixed parameter values. However, very few have investigated why such parameter control approaches perform better. In principle, it could be the case that using different parameter values alone is already sufficient and EA performance can be improved without sophisticated control strategies raising an issue in the methodology of parameter control mechanisms' evaluation. This paper investigates whether very simple random variation in parameter values during an evolutionary run can already provide improvements over static values. Results suggest that random variation of parameters should be included in the benchmarks when evaluating a new parameter control mechanism.

This paper is concerned with \textit{on-line} evolutionary robotics, where robot controllers are being evolved during a robots' operative time. This approach offers the ability to cope with environmental changes without human intervention, but to be effective it needs an automatic parameter control mechanism to adjust the evolutionary algorithm (EA) appropriately. In particular, mutation step sizes ($\sigma$) and the time spent on fitness evaluation ($\tau$) have a strong influence on the performance of an EA. In this paper, we introduce and experimentally validate a novel method for self-adapting $\tau$ during runtime. The results show that this mechanism is viable: the EA using this self-adaptative control scheme consistently shows decent performance without a priori tuning or human intervention during a run.

Evolution can be employed for two goals. Firstly, to provide a force for adaptation to the environment as it does in nature and in many artificial life implementations - this allows the evolving population to survive. Secondly, evolution can provide a force for optimisation as is mostly seen in evolutionary robotics research - this causes the robots to do something useful. We propose the MONEE algorithmic framework as an approach to combine these two facets of evolution: to combine environment-driven and task-driven evolution. To achieve this, MONEE employs environment-driven and task-based parent selection schemes in parallel.

In evolutionary robotics, robot controllers are often evolved in a separate development phase preceding actual deployment - we call this off-line evolution. In on-line evolutionary robotics, by contrast, robot controllers adapt through evolution while the robots perform their proper tasks, not in a separate preliminary phase. In this case, individual robots can contain their own self-sufficient evolutionary algorithm (the encapsulated approach) where individuals are typically evaluated by means of a time sharing scheme: an individual is given the run of the robot for some amount of time and fitness corresponds to the robot's task performance in that period.

In this paper we introduce the use of Scalable Vector Graphics (SVG) as a representation for evolutionary art. We describe the technical aspects of using SVG in evolutionary art, and explain the genetic operators mutation and crossover. Furthermore, we compare the use of SVG with existing representations in evolutionary art. We performed a number of experiments in an unsupervised evolutionary art system using two aesthetic measures as fitness functions, and compared the outcome of the different experiments with each other and with previous work with symbolic expressions as the representation. All images and SVG code examples in this paper are available at http://www.few.vu.nl/~eelco

This paper presents part of an endeavour towards robots and robot collectives that can adapt their controllers autonomously and self-sufficiently and so independently learn to cope with situations unforeseen by their designers. We introduce the Embodied Distributed Evolutionary Algorithm (DEA) for on-line, on-board adaptation of robot controllers. We experimentally evaluate DEA using a number of well-known tasks in the evolutionary robotics field to determine whether it is a viable implementation of on-line, on-board evolution. We compare it to the encapsulated mu + 1 ON- LINE algorithm in terms of (the stability of) task performance and the sensitivity to parameter settings.

In this paper we investigate how life expectation influences the development of risk attitude within an artificial species. Our hypothesis is that agents with a very long life span are likely to become more risk averse because they have more to lose. To assess this hypothesis we set up a simple system, based on Sugarscape, where risk attitude is an inheritable (hence, evolvable) property. Performing numerous simulations with different versions of this system we found that long-lived agents consistently and clearly evolve a more risk averse behavior than short-lived agents. Perceiving evolution as a general force towards optimal behavior, these results indicate that increased risk avoidance is a generally good strategy for agents with a higher life expectation. This finding can be used to explain various real-world phenomena. For instance, it can clarify the fact that people tend to adopt risky strategies when their life is in danger.

This paper describes and experimentally evaluates the viability of the (μ + 1) ON-LINE evolutionary algorithm for on-line adaptation of robot controllers. Secondly, it explores the parameter space for this algorithm and identifies four important parameters: the population size μ, the re-evaluation rate ρ, the mutation step-size σ and the controller evaluation period τ. Subsequently, it investigates their influence on controller performance, stability of behaviour and speed of adaptation. The results indicate that the encapsulated on-line evolutionary approach is a viable one and merits further research. In agreement with existing research, the mutation step-size σ proves to be of overriding importance to finding good solutions. Specific to on-line evolution, the results show that longer evaluation times greatly benefit the quality of controllers as well as stability of behaviour and speed of adaptation.

In this paper we investigate and compare three aesthetic measures within the context of evolutionary art. We evolve visual art with an unsupervised evolutionary art system using genetic programming and an aesthetic measure as the fitness function. We perform multiple experiments with different aesthetic measures and examine their influence on the evolved images. Additionally, we perform a cross-evaluation by calculating the aesthetic value of images evolved by measure i according to measure j. This way we investigate the flexiblity of each aesthetic measure (i.e., whether the aesthetic measure appreciates different types of images). Last, we perform an image analysis using a fixed set of image statistics functions. The results show that aesthetic measures have a rather clear `style' and that these styles can be very different. Furthermore we find that some aesthetic measures show little flexibility and appreciate only a limited set of images. The images in this paper might only be in color in the electronic version.

We present a case study demonstrating that using the REVAC parameter tuning method we can greatly improve the `world champion' EA (the winner of the CEC-2005 competition) with little effort. For `normal' EAs the margins for possible improvements are likely much bigger. Thus, the main message of this paper is that using REVAC great performance improvements are possible for many EAs at moderate costs. Our experiments also disclose the existence of `specialized generalists', that is, EAs that are generally good on a set of test problems, but only w.r.t. one performance measure and not along another one. This shows that the notion of robust parameters is questionable and the issue requires further research. Finally, the results raise the question what the outcome of the CEC-2005 competition would have been, if all of EAs had been tuned by REVAC, but without further research it remains an open question whether we crowned the wrong king.

In this paper we discuss the notion of situated evolution. Our treatment includes positioning situated evolution on the map of evolutionary processes in terms of time- and space-embeddedness, and the identification of decentralization as an orthogonal property. We proceed with a selected overview of related literature in the categories of our interest. This overview enables us to distill further details that distinguish the encountered methods. As it turns out the essential differences can be captured through the mechanics of selection and fertilization. These insights are aggregated into a new model called the situated evolution method, which is then used to provide a fine-grained map of existing work.

Tuning the parameters of an evolutionary algorithm (EA) to a given problem at hand is essential for good algorithm performance. Optimizing parameter values is, however, a non-trivial problem, beyond the limits of human problem solving.In this light it is odd that no parameter tuning algorithms are used widely in evolutionary computing. This paper is meant to be stepping stone towards a better practice by discussing the most important issues related to tuning EA parameters, describing a number of existing tuning methods, and presenting a modest experimental comparison among them. The paper is concluded by suggestions for future research - hopefully inspiring fellow researchers for further work.

In this paper the effects of individual learning on an evolving population of situated agents are investigated. We work with a novel type of system where agents can decide autonomously (by their controllers) if/when they reproduce and the bias in the agent controllers for the mating action is adaptable by individual learning. Our experiments show that in such a system reinforcement learning with the straightforward rewards system based on energy makes the agents lose their interest in mating. In other words, we see that learning frustrates evolution, killing the whole population on the long run. This effect can be counteracted by introducing a specially designated positive mating reward, pretty much like an orgasm in Nature. With this twist individual learning becomes a positive force. It can make the otherwise disappearing population viable by keeping agents alive that did not yet learn the task at hand. This hiding effect proves positive for it provides a smooth road for the population to adapt and learn the task with a lower risk of extinction.

The subject of the present investigation is population-based adaptive systems (PAS), as implemented in the NEW TIES platform. In many existing PASs two adaptation mechanisms are combined, (non-Lamarckian) evolution and individual learning, inevitably raising the issue of dasiaforgetful populationspsila: individually learned knowledge disappears when the individual that learned it dies. We propose social learning by explicit knowledge transfer to overcome this problem. Our mechanism is based on 1) direct communication among agents in the population, 2) messages carrying rules that the sender agent uses in its controller, and 3) the ability of the recipient agent to incorporate foreign rules into its controller. Thus, knowledge can be disseminated and multiplied within the same generation, making the population a knowledge reservoir for individually acquired knowledge. We present an initial assessment of this idea and show that this social mechanism is capable of efficiently distributing knowledge and improving the performance of the population.

Calibrating an evolutionary algorithm (EA) means finding the right values of algorithm parameters for a given problem. This issue is highly relevant, because it has a high impact (the performance of EAs does depend on appropriate parameter values), and it occurs frequently (parameter values must be set before all EA runs). This issue is also highly challenging, because finding good parameter values is a difficult task. In this paper we propose an algorithmic approach to EA calibration by describing a method, called REVAC, that can determine good parameter values in an automated manner on any given problem instance. We validate this method by comparing it with the conventional hand-based calibration and another algorithmic approach based on the classical meta-GA. Comparative experiments on a set of randomly generated problem instances with various levels of multi-modality show that GAs calibrated with REVAC can outperform those calibrated by hand and by the meta-GA.

Calibrating the parameters of an evolutionary algorithm (EA) is a laborious task. The highly stochastic nature of an EA typically leads to a high variance of the measurements. The standard statistical method to reduce variance is measurement replication, i.e., averaging over several test runs with identical parameter settings. The computational cost of measurement replication scales with the variance and is often too high to allow for results of statistical significance. In this paper we study an alternative: the REVAC method for Relevance Estimation and Value Calibration, and we investigate how different levels of measurement replication influence the cost and quality of its calibration results. Two sets ofof experiments are reported: calibrating a genetic algorithm on standard benchmark problems, and calibrating a complex simulation in evolutionary agent-based economics. We find that measurement replication is not essential to REVAC, which emerges as a strong and efficient alternative to existing statistical methods.

This paper introduces the collective neuro evolution (CONE) method, and compares its efficacy for designing specialization, with a conventional neuro-evolution (NE) method. Specialization was defined at both the individual agent, and at the agent group level. The CONE method was tested comparatively with the conventional NE method in an extension of the multi-rover task domain, where specialization exhibited at both the individual and group level is known to benefit task performance. In the multi-rover domain, the task was for many agents (rovers) to maximize the detection and evaluation of points of interest in a simulated environment, and to communicate gathered information to a base station. The goal of the rover group was to maximize a global evaluation function that measured performance (fitness) of the group. Results indicate that the CONE method was appropriate for facilitating specialization at both the individual and agent group levels, where as, the conventional NE method succeeded only in facilitating individual specialization. As a consequence of emergent specialization derived at both the individual and group levels, rover groups evolved by the CONE method were able to achieve a significantly higher task performance, comparative to groups evolved by the conventional NE method.

No abstract available.

This work introduces Autonomous selection in EAs to escape the need for some central control during the selection phases of an EA. The results demonstrate that this is a viable idea that needs further investigation.

We study the benefit of measurement replication when using the Relevance Estimation and Value Calibration method to calibrate a genetic algorithm. We find that replication is not essential to REVAC, which makes it a strong alternative to existing statistical tools which are computationally costly.

Neuro-Evolution (NE) methods have been successfully applied to the rover task domain [1], [2]. However, extending this domain to include the notion of using NE to facilitate emergent specialization, in order to increase task performance, has not yet been investigated. We introduce the Collective Neuro Evolution (CONE) method, and compares its efficacy for designing specialization, with a conventional NE method. CONE and conventional NE were applied to an extension of the multi-rover task [1], [2] for the purpose of designing collective behavior. This task requires solutions for controlling groups of simulated autonomous vehicles (rovers) that seek to maximize the number of points of interest discovered in an unexplored environment (global evaluation function). Rovers operated in a discrete simulation environment, and used complementary sensors and actuators so as to maximize the global evaluation function.

In this paper we describe and evaluate a fully distributed P2P evolutionary algorithm (EA) with adaptive autonomous selection. Autonomous selection means that decisions regarding survival and reproduction are taken by the individuals themselves independently, without any central control.This allows for a fully distributed EA, where not only reproduction (crossover and mutation) but also selection is performed at local level. An unwanted consequence of adding and removing individuals in a non-synchronized manner is that the population size gets out of control too. This problem is resolved by addingan adaptation mechanism allowing individuals to regulate their own selection pressure. The key tothis is a gossiping algorithm that enables individuals to maintain estimates on the size andthe fitness of the population. The algorithm is experimentally evaluated on a test problem to show the viability of the idea and to gain insight into the run-time dynamics of such an algorithm. The results convincingly demonstrate the feasibility of a fully decentralized EA in which the population size can be kept stable.

In this research, a neuro-evolution method called collective neuro-evolution (CONE), is introduced for the design of neural controllers (agents) operating in collective behavior task domains. The efficacy of the CONE method for facilitating emergent behavioral specialization for the benefit of increasing task performance is tested in a pursuit-evasion and collective gathering task. For a comparative study, a conventional neuro-evolution method was applied to the same tasks. In both tasks, the CONE method derived behavioral specialization in groups of agents resulting in higher task performances, where as the conventional neuro-evolution method was unable to derive specialization resulting in comparatively lower task performances

To analyze emergent behavior, a formal framework is needed to characterize the structure and dynamics of complex interaction-based multi-agent systems. We introduce an extension of an existing agent testbed for artificial societies making it possible to formally analyze the dynamics of the simulated agent system. The extension generates temporally annotated logical terms that describe parts of the dynamics of the simulated system. Based on these terms, it is possible to validate hypotheses about the system on different levels of aggregation, i.e. agent, group and system level. We present first results from a set of simple experiments in a class of artificial societies.

In this paper we evaluate a new approach to selection in genetic algorithms (GAs). The basis of our approach is that the selection pressure is not a superimposed parameter defined by the user or some Boltzmann mechanism. Rather, it is an aggregated parameter that is determined collectively by the individuals in the population. We implement this idea in two different ways and experimentally evaluate the resulting genetic algorithms on a range of fitness landscapes. We observe that this new style of selection can lead to 30-40% performance increase in terms of speed.

We present and evaluate a method for estimating the relevance and calibrating the values of parameters of an evolutionary algorithm. The method provides an information theoretic measure on how sensitive a parameter is to the choice of its value. This can be used to estimate the relevance of parameters, to choose between different possible sets of parameters, and to allocate resources to the calibration of relevant parameters. The method calibrates the evolutionary algorithm to reach a high performance, while retaining a maximum of robustness and generalizability. We demonstrate the method on an agent-based application from evolutionary economics and show how the method helps to design an evolutionary algorithm that allows the agents to achieve a high welfare with a minimum of algorithmic complexity.

In this paper we study a population of individuals in a simulated artificial environment. These individuals have a "body" as well as a "mind", i.e., some of their features effect their "physical" properties, like speed and strength, while other features influence their "mental" preferences and choices in interacting with the environment and other agents. We compare two approaches to adapting the minds of individuals. In approach 1, the bodies and the minds develop through evolution, while in approach 2 only the bodies evolve and the minds are adapted by lifetime-learning. The results indicate that the evolutionary approach is able to sustain larger and more stable agent populations as well as maintain a higher degree of individual success compared to the lifetime learning approach. Furthermore, quite unexpectedly, the method used for mental development has a strong effect on the development of the physical features within the very same environment: The individuals' bodies evolve to completely different segments of the physical feature space under the two regimes.

We study a selected group of hybrid EAs for solving CSPs, consisting of the best performing EAs from the literature. We investigate the contribution of the evolutionary component to their performance by comparing the hybrid EAs with their "de-evolutionarised" variants. The experiments show that "de-evolutionarising" can increase performance, in some cases doubling it. Considering that the problem domain and the algorithms are arbitrarily selected from the "memetic niche", it seems likely that the same effect occurs for other problems and algorithms. Therefore, our conclusion is that after designing and building a memetic algorithm, one should perform a verification by comparing this algorithm with its "de-evolutionarised" variant.

The research goal was to engineer agent collectives that most effectively accomplish a cooperative gathering task. In view of this, we compared reproduction schemes for the artificial evolution of agent controller parameters for a cooperative minesweeping task. Agents utilized cooperative behavior to improve task performance in a simulated environment where different types of mines with different fitness rewards were randomly distributed. We compared the evolution of agent controller parameters with respect to temporal and spatial dimensions of agent reproduction schemes. The first dimension concerned agents reproducing only once at the end of their lifetime or multiple times during their lifetime. The second dimension concerned agents reproducing only with agents in adjacent positions (locally restricted) or with agents located anywhere else in the environment (panmictic). Results indicated that the single reproduction at the end of an agent's lifetime and the locally restricted reproduction schemes afforded the agent collective a higher level of performance in its cooperative gathering task

This paper investigates the effects of two different communication protocols within an artificial society, where communication and cooperation is necessary to survive. Communication in our system is not a hard-coded behavior, rather it is an evolvable feature. The two protocols we consider differ significantly. Using the first approach, individuals multicast messages that can be received by any individual. In the second approach, based on the so-called newscast computing model, individuals send a message to their list of ”friends ” only, where this list is frequently updated. These protocols are compared experimentally by their effects on population dynamics and the evolution of communicativeness. The results provide new insights into the niche of newscast-based communication protocols: we identify two essential processes (information being spread and information loosing its value) and consider the ratio of the speeds of these processes as a basic indicator for communication success.

This work reports on experiments with an artificial society simulation package. This work is part of a larger project whose main goal is to investigate the emergence of cooperation and communication in response of (scalable) environmental challenges. The specific goals of the experiments reported here include: 1) the study a number of extensions of the classical SugarScape model; 2) to compare two radically different approaches to communication among the individuals of the population. Our results demonstrate that a number of the presented extensions should be taken up in future experiments in artificial societies, and that the decentralised communication protocol has negative effects on the system behaviour.

Constraint handling is not straightforward in evolutionary algorithms (EAs) since the usual search operators, mutation and recombination, are 'blind' to constraints. Nevertheless, the issue is highly relevant, for many challenging problems involve constraints. Over the last decade, numerous EAs for solving constraint satisfaction problems (CSP) have been introduced and studied on various problems. The diversity of approaches and the variety of problems used to study the resulting algorithms prevents a fair and accurate comparison of these algorithms. This paper aligns related work by presenting a concise overview and an extensive performance comparison of all these EAs on a systematically generated test suite of random binary CSPs. The random problem instance generator is based on a theoretical model that fixes deficiencies of models and respective generators that have been formerly used in the evolutionary computing field.

Evolutionary approaches to constraint satisfaction problems (CSPs) are often tested on a set of randomly generated instances. Recently it has been shown that the frequently used random problem instance generators are not good enough. This implies that conclusions based on the usage of these generators need to be revised. In this paper, we perform an extensive experimental comparison of CSP solving EAs based on a new, improved generator. In particular, we compare two representations, two evaluation functions and EAs with and without the so-called SAWing mechanism, each of these with different population sizes and variation operators. The resulting systematic overview confirms some "myths", while refutes others. Most notably, the role of the population size and that of SAWing itself turn out to be different from what is usually assumed.

In this paper, we point to some essential shortcomings in contemporary practice in performing and documenting experimental research in evolutionary computing (EC). We identify some crucial problems and the limitations of this practice, and elaborate on research directions that should be pursued to improve the quality and relevance of experimental research.

Evolutionary algorithms (EAs) for solving constraint satisfaction problems (CSPs) can be roughly divided into two classes: EAs with adaptive fitness functions and heuristic-based EAs. A.E. Eiben et al. (1998) compared effective EAs of the first class experimentally using a large set of benchmark instances consisting of randomly-generated binary CSPs. In this paper, we complete this comparison by performing the same experiments using three of the most effective heuristic-based EAs. The results of our experiments indicate that the three heuristic-based EAs have similar performances on random binary CSPs. Comparing these results with those of A.E. Eiben et al., we are able to identify the best EA for binary CSPs as the algorithm introduced by G. Dozier et al. (1994), which uses a heuristic as well as an adaptive fitness function.

This paper describes a project funded by the European Commission which seeks to provide the technology and software infrastructure necessary to support the next generation of evolving infohabitants in a way that makes that infrastructure universal, open and scalable. The Distributed Resource Evolutionary Algorithm Machine (DREAM) will use existing hardware infrastructure in a more efficient manner, by utilising otherwise unused CPU time. It will allow infohabitants to co-operate, communicate, negotiate and trade; and emergent behaviour is expected to result. It is expected that there will be an emergent economy that results from the provision and use of CPU cycles by infohabitants and their owners. The DREAM infrastructure will be evaluated with new work on distributed data mining, distributed scheduling and the modelling of economic and social behaviour.

In this paper two different generalizations of intermediate recombination in evolution strategies are investigated. Both generalizations allow for recombining an arbitrary number of /spl rho/ parents. However, the so-called /spl rho///spl rho/-mechanism averages all /spl rho/ parents, while the so-called /spl rho//2-mechanism repeatedly (for each object variable anew) selects two out of /spl rho/ parents and averages the corresponding object variables to create an offspring individual. Results presented for the spherical function demonstrate that these two operators can cause a significantly different behavior concerning the convergence velocity of the algorithm. Both operators are applied to a number of different objective functions (including separable and non-separable, unimodal and multimodal, regular and irregular topologies), and the impact of the number of parents /spl rho/ is investigated. The results illustrate that important differences in the results are not consistent with the canonical topology classification of objective functions, but can be explained to some extent by the "genetic repair" hypothesis of Beyer in combination with a reasoning about the success region.

Several papers have reported experimental results for multiparent recombination operators, looking at the effects of using more parents. Tacitly, these studies assume that the number of parents (the arity of the given recombination operator) tells how many old individuals contribute to a new one by passing their genetic information to it. We point out that this assumption is not valid for a number of well-known recombination operators and distinguish parents and donors, the latter being those parents that really deliver information to the offspring. We perform a mainly theoretical analysis on the number of donors. We also consider the mechanisms for choosing the alleles for a child and find indications for a too strong genetic drift for occurrence-based scanning. Experimental results are provided to support theoretical estimates and predictions.

The issue of controlling values of various parameters of an evolutionary algorithm is one of the most important and promising areas of research in evolutionary computation: it has a potential of adjusting the algorithm to the problem while solving the problem. In the paper we: 1) revise the terminology, which is unclear and confusing, thereby providing a classification of such control mechanisms, and 2) survey various forms of control which have been studied by the evolutionary computation community in recent years. Our classification covers the major forms of parameter control in evolutionary computation and suggests some directions for further research.

Exploration and exploitation are the two cornerstones of problem solving by search. The common opinion about evolutionary algorithms is that they explore the search space by the (genetic) search operators, while exploitation is done by selection. Thi

An extension of evolution strategies to multiparent recombination involving a variable number [symbol: see text] of parents to create an offspring individual is proposed. The extension is experimentally evaluated on a test suite of functions differing in their modality and separability and the regular/irregular arrangement of their local optima. Multiparent diagonal crossover and uniform scanning crossover and a multiparent version of intermediary recombination are considered in the experiments. The performance of the algorithm is observed to depend on the particular combination of recombination operator and objective function. In most of the cases a significant increase in performance is observed as the number of parents increases. However, there might also be no significant impact of recombination at all, and for one of the unimodal objective functions, the performance is observed to deteriorate over the course of evolution for certain choices of the recombination operator and the number of parents. Additional experiments with a skewed initialization of the population clarify that intermediary recombination does not cause a search bias toward the origin of the coordinate system in the case of domains of variables that are symmetric around zero.