On the path towards truly autonomous robots, embodied agents will require to be adaptable to unforeseen circumstances. Yet, most robotic agents still suffer from significant performance degradation when scenarios change slightly, with many even failing their tasks entirely. In contrast, organisms in nature exhibit strong adaptability, largely due to bio-diversity, which has prevented the extinction of life throughout severe environmental changes. The concept of quality-diversity aims to emulate this natural resilience, yielding robust results through diversification of embodied agents in the behavior space. However, in nature, diversity occurs simultaneously at multiple levels: body, brain, and behavior. This study on the body-brain optimization of virtual embodied agents spans two brain representations—an Artificial Neural Network (ANN) and a graph—and investigates these levels to determine the most critical scope for diversity in fostering performance, generality, and robustness. We start by optimizing for a simple locomotion task, and then evaluate generality through transfer to a diverse set of tasks, including locomotion in new environments and interaction with objects. Our findings confirm the importance of simultaneously considering multiple axes of diversity for achieving good performance and adaptability—demonstrating zero-shot transfer on 18 new tasks. Moreover, we observe that the graph controller performs on par with the ANN, offering greater interpretability.

In complex control tasks, finding high-performing policies often requires discovering and exploiting specific behavioral strategies. While Quality-Diversity (QD) algorithms can uncover these strategies through extensive behavior space exploration, they sacrifice efficiency by improving suboptimal behaviors. Conversely, Evolution Strategies (ES) achieve impressive performance through focused optimization but frequently become trapped in local optima due to limited behavioral exploration. We present Quality with J ust E nough Di versity (JEDi), a new optimization framework that resolves this fundamental tension. JEDi employs a combination of Gaussian Process modeling and parallel ES to intelligently explore behavioral space while maintaining focused optimization. At its core, JEDi learns a probabilistic mapping between behaviors and performance, using this model to identify and target promising behavioral regions that could unlock better solutions. This targeted exploration is achieved through multiple Evolution Strategy emitters that simultaneously optimize toward selected behaviors while maximizing task performance. To further improve JEDi’s exploration capabilities, we introduce its Dynamic variant DyJEDi with an adaptive restart mechanism that dynamically detects and responds to emitter convergence, independently restarting each Evolution Strategy when it stagnates in both behavior and fitness space. This dynamic approach significantly improves exploration efficiency and robustness to local optima. We demonstrate that DyJEDi outperforms both traditional ES and QD approaches across challenging continuous robotics control tasks, achieving higher final performance. Most notably, DyJEDi solves several hard exploration problems where standard ES methods consistently fail.

Vision-based decision-making tasks encompass a wide range of applications, including safety-critical domains where trustworthiness is as key as performance. These tasks are often addressed using Deep Reinforcement Learning (DRL) techniques, based on Artificial Neural Networks (ANNs), to automate sequential decision making. However, the "black-box" nature of ANNs limits their applicability in these settings, where transparency and accountability are essential. To address this, various explanation methods have been proposed; however, they often fall short in fully elucidating the decision-making pipeline of ANNs, a critical aspect for ensuring reliability in safety-critical applications. To bridge this gap, we propose an approach based on Graph-based Genetic Programming (GGP) to generate transparent policies for vision-based control tasks. Our evolved policies are constrained in size and composed of simple and well-understood operational modules, enabling inherent interpretability. We evaluate our method on three Atari games, comparing explanations derived from common explainability techniques to those derived from interpreting the agent's true computational graph. We demonstrate that interpretable policies offer a more complete view of the decision process than explainability methods, enabling a full comprehension of competitive game-playing policies.

Semantic segmentation models trained on synthetic data often perform poorly on real-world images due to domain gaps, particularly in adverse conditions where labeled data is scarce. Yet, recent foundation models enable to generate realistic images without any training. This paper proposes to leverage such diffusion models to improve the performance of vision models when learned on synthetic data. We introduce two novel techniques for semantically consistent style transfer using diffusion models: Class-wise Adaptive Instance Normalization and Cross-Attention ( CACTI ) and its extension with selective attention Filtering ( CACTI F ). CACTI applies statistical normalization selectively based on semantic classes, while CACTI F further filters cross-attention maps based on feature similarity, preventing artifacts in regions with weak cross-attention correspondences. Our methods transfer style characteristics while preserving semantic boundaries and structural coherence, unlike approaches that apply global transformations or generate content without constraints. Experiments using GTA5 as source and Cityscapes/ACDC as target domains show that our approach produces higher quality images with lower FID scores and better content preservation. Our work demonstrates that class-aware diffusion-based style transfer effectively bridges the synthetic-to-real domain gap even with minimal target domain data, advancing robust perception systems for challenging real-world applications. The source code is available at: https://github.com/echigot/cactif .

In evolutionary policy search, neural networks are usually represented using a direct mapping: each gene encodes one network weight. Indirect encoding methods, where each gene can encode for multiple weights, shorten the genome to reduce the dimensions of the search space and better exploit permutations and symmetries. The Geometric Encoding for Neural network Evolution (GENE) introduced an indirect encoding where the weight of a connection is computed as the (pseudo-)distance between the two linked neurons, leading to a genome size growing linearly with the number of genes instead of quadratically in direct encoding. However GENE still relies on hand -crafted distance functions with no prior optimization. Here we show that better performing distance functions can be found for GENE using Cartesian Genetic Programming (CGP) in a meta-evolution approach, hence optimizing the encoding to create a search space that is easier to exploit. We show that GENE with a learned function can outperform both direct encoding and the hand-crafted distances, generalizing on unseen problems, and we study how the encoding impacts neural network properties.

Evolutionary Algorithms (EA) have been successfully used for the optimization of neural networks for policy search, but they still remain sample inefficient and underperforming in some cases compared to gradient-based reinforcement learning (RL). Various methods combine the two approaches, many of them training a RL algorithm on data from EA evaluations and injecting the RL actor into the EA population. However, when using Evolution Strategies (ES) as the EA, the RL actor can drift genetically far from the the ES distribution and injection can cause a collapse of the ES performance. Here, we highlight the phenomenon of genetic drift where the actor genome and the ES population distribution progressively drift apart, leading to injection having a negative impact on the ES. We introduce Genetic Drift Regularization (GDR)11https://anonymous.4open.science/r/GDR-E784, a simple regularization method in the actor training loss that prevents the actor genome from drifting away from the ES. We show that GDR can improve ES convergence on problems where RL learns well, but also helps RL training on other tasks, fixes the injection issues better than previous controlled injection methods

Graph-based Genetic Programming (GGP) can create interpretable control policies in graph form, but faces challenges such as local optima and solution fragility, which undermine its efficacy. Quality-Diversity (QD) has been effective in addressing similar issues, traditionally in Artificial Neural Network (ANN) optimization. In this paper, we introduce a general Graph Quality-Diversity (G-QD) framework to enhance the performance of GGP with QD optimization, obtaining a variety of interpretable, effective, and resilient policies. Using Cartesian Genetic Programming (CGP) as the GGP technique and MAP-Elites (ME) as the QD algorithm, we leverage a combination of behavior and graph structural descriptors. Experimenting on two navigation and two locomotion continuous control tasks, our framework yields an array of effective yet behaviorally and structurally diverse policies, surpassing the performance of a standard Genetic Algorithm (GA). The resulting solution set also increases interpretability, allowing for insight into the control tasks. Additionally, our experiments demonstrate the robustness of the solutions to faults such as sensor damage.

Evolution Strategies (ES) are effective gradient-free optimization methods that can be competitive with gradient-based approaches for policy search. ES only rely on the total episodic scores of solutions in their population, from which they estimate fitness gradients for their update with no access to true gradient information. However this makes them sensitive to deceptive fitness landscapes, and they tend to only explore one way to solve a problem. Quality-Diversity methods such as MAP-Elites introduced additional information with behavior descriptors (BD) to return a population of diverse solutions, which helps exploration but leads to a large part of the evaluation budget not being focused on finding the best performing solution. Here we show that behavior information can also be leveraged to find the best policy by identifying promising search areas which can then be efficiently explored with ES. We introduce the framework of Quality with Just Enough Diversity (JEDi) which learns the relationship between behavior and fitness to focus evaluations on solutions that matter. When trying to reach higher fitness values, JEDi outperforms both QD and ES methods on hard exploration tasks like mazes and on complex control problems with large policies.

Critical software systems require stringent testing to identify possible failure cases, which can be difficult to find using manual testing. In this study, we report our industrial experience in testing a realistic R&D flight control system using a heuristic based testing method. Our approach utilizes evolutionary strategies augmented with intrinsic motivation to yield a diverse range of test cases, each revealing different potential failure scenarios within the system. This diversity allows for a more comprehensive identification and understanding of the system’s vulnerabilities. We analyze the test cases found by evolution to identify the system’s weaknesses. The results of our study show that our approach can be used to improve the reliability and robustness of avionics systems by providing high-quality test cases in an efficient and cost-effective manner.

The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.

When searching for policies, reward-sparse environments often lack sufficient information about which behaviors to improve upon or avoid. In such environments, the policy search process is bound to blindly search for reward-yielding transitions and no early reward can bias this search in one direction or another. A way to overcome this is to use intrinsic motivation in order to explore new transitions until a reward is found. In this work, we use a recently proposed definition of intrinsic motivation, Curiosity, in an evolutionary policy search method. We propose Curiosity-ES, 1 an evolutionary strategy adapted to use Curiosity as a fitness metric. We compare Curiosity-ES with other evolutionary algorithms intended for exploration, as well as with Curiosity-based reinforcement learning, and find that Curiosity-ES can generate higher diversity without the need for an explicit diversity criterion and leads to more policies which find reward.

Selection in genetic algorithms is difficult for stochastic problems due to noise in the fitness space. Common methods to deal with this fitness noise include sampling multiple fitness values, which can be expensive. We propose LUCIE, the Lower Upper Confidence Intervals Elitism method, which selects individuals based on confidence. By focusing evaluation on separating promising individuals from others, we demonstrate that LUCIE can be effectively used as an elitism mechanism in genetic algorithms. We provide a theoretical analysis on the convergence of LUCIE and demonstrate its ability to select fit individuals across multiple types of noise on the OneMax and LeadingOnes problems. We also evaluate LUCIE as a selection method for neuroevolution on control policies with stochastic fitness values.

A major limitation to the optimization of artificial neural networks (ANN) with evolutionary methods lies in the high dimensionality of the search space, the number of weights growing quadratically with the size of the network. This leads to expensive training costs, especially in evolution strategies which rely on matrices whose sizes grow with the number of genes. We introduce a geometric encoding for neural network evolution (GENE) as a representation of ANN parameters in a smaller space that scales linearly with the number of neurons, allowing for efficient parameter search. Each neuron of the network is encoded as a point in a latent space and the weight of a connection between two neurons is computed as the distance between them. The coordinates of all neurons are then optimized with evolution strategies in a reduced search space while not limiting network fitness and possibly improving search.

Cartesian Genetic Programming (CGP) has previously shown capabilities in image processing tasks by evolving programs with a function set specialized for computer vision. A similar approach can be applied to Atari playing. Programs are evolved using mixed type CGP with a function set suited for matrix operations, including image processing, but allowing for controller behavior to emerge. While the programs are relatively small, many controllers are competitive with state of the art methods for the Atari benchmark set and require less training time. By evaluating the programs of the best evolved individuals, simple but effective strategies can be found.

Genetic Regulatory Networks (GRNs) implementations have a high degree of variability in their details. Parameters, encoding methods, and dynamics formulas all differ in the literature, and some GRN implementations have a high degree of model complexity. In this paper, we present a comparative study of different implementations of a GRN and introduce new variants for comparison. We use a modified Genetic Algorithm (GA) to evaluate GRN performance on a number of common benchmark tasks, with a focus on real-time control problems. We propose an encoding scheme and set of dynamics equations that simplifies implementation and evaluate the evolutionary fitness of this proposed method. Lastly, we use the comparative modifications study to demonstrate overall enhancements for GRN models.

We present DevoII, an improved cell-based developmental model for wind farm layout optimization. To address the shortcomings of discretization, DevoII's gene regulatory networks control cells that act in a continuous rather than discretized grid space. We find that DevoII is competitive, and in some cases, superior with respect to state-of-the-art global, stochastic search approaches when a suite of algorithms is evaluated on different wind scenarios. The modularity of the genetic regulatory network computational paradigm in terms of isolating its search algorithm, the regulatory network simulation and the cell simulation, allowed this improvement to largely focus upon cell simulation. This indicates a robustness property of the paradigm's design. As well, wflo highlights how developmental models can be considered more efficient than other optimization methods because of their "optimize once, use-many" adaptability.

Optimizing a wind farm layout is a very complex problem that involves many local and global constraints such as inter-turbine wind interference or terrain peculiarities. Existing methods are either inefficient or, when efficient, take days or weeks to execute. Solutions are contextually sensitive to the specific values of the problem variables; when one value is modified, the algorithm has to be re-run from scratch. This paper proposes the use of a developmental model to generate farm layouts. Controlled by a gene regulatory network, virtual cells have to populate a simulated environment that represents the wind farm. When the cells' behavior is learned, this approach has the advantage that it is re-usable in different contexts; the same initial cell is responsive to a variety of environments and the layout generation takes few minutes instead of days.