Most prior works on communication in multi-agent reinforcement learning have focused on emergent communication, which often results in inefficient and non-interpretable systems. Inspired by the role of language in natural intelligence, we investigate how grounding agents in a human-defined language can improve the learning and coordination of embodied agents. We propose a framework in which agents are trained not only to act but also to produce and interpret natural language descriptions of their observations. This language-augmented learning serves a dual role: enabling efficient and interpretable communication between agents, and guiding representation learning. We demonstrate that language-augmented agents outperform emergent communication baselines across various tasks. Our analysis reveals that language grounding leads to more informative internal representations, better generalization to new partners, and improved capability for human-agent interaction. These findings demonstrate the effectiveness of integrating structured language into multi-agent learning and open avenues for more interpretable and capable multi-agent systems.1 1 Appendices and code are available at github.com/MToquebiau/LAMARL

In this work, we explore the evolution of neural controllers to coordinate a swarm of robotic agents that dynamically adjust their state to match a target pattern defined at the macroscopic level (e.g. each robot should display a specific color so that an external observer sees a coherent picture from the swarm). Inspired by the multi-cellular flag problem, we compare static and dynamic swarm setups using a sliding puzzle-inspired grid environment. We use evolutionary learning to optimize artificial neural network (ANN) controllers that regulate agent behaviors based on local communication. We analyze the impact of agent density and movement fluidity on pattern formation, and we demonstrate that allowing controlled movement can enhance adaptability while preserving global structure. Post-mortem analyses reveal key differences in learned strategies between static and dynamic configurations regarding generalization to varying swarm density, including validation with real robots. We also reveal how communication speed is critical when the swarm configuration changes over time.

This paper deals with policy learning for a team of heterogeneous robotic agents when the whole team shares a single reward. We address the problem of providing an accurate estimation of the contribution of each agent in tasks where coordination between agents requires joint policy updates of two (or more) agents. This is typically the case when two agents must simultaneously modify their behaviors to perform a joint action that leads to a performance gain for the whole team. We propose a cooperative co-evolutionary algorithm extended with a multi-armed bandit algorithm that dynamically adjusts the number of agents that should update their policies simultaneously, aiming both for performance and learning speed. We use a realistic robotic multi-rover task where agents must physically distribute themselves on points of interest of different natures to complete the task. Results show that the algorithm is able to select the best group size for policy updates that reflects the task's coordination requirements. Surprisingly, we also reveal that coupling between agents' actions in a realistic setup can also emerge from interactions at the phenotypical level, hinting at subtle interactions during learning between the control parameter space and the behavioral space.

In this paper, we explore how robots in a swarm can individually exploit collisions to produce self-organizing behaviours at the macroscopic scale. We propose to focus on two behaviours that modify the orientation of a robot during a collision, which are inspired by positive and negative feedback observed in Nature. These two behaviours differ in the nature of the feedback produced after a collision by favouring either (1) the alignment or (2) the anti-alignment of the robot with an external force, whether it is an obstacle or another robot. We describe a social learning algorithm using evolutionary operators to learn individual policies that exploit these behaviours in an online and distributed fashion. This algorithm is validated both in simulation and with real robots to solve two tasks involving phototaxis, one of which requires self-organized aggregation to be completed.

State-of-the-art reinforcement learning is now able to learn versatile locomotion, balancing and push-recovery capabilities for bipedal robots in simulation. Yet, the reality gap has mostly been overlooked and the simulated results hardly transfer to real hardware. Either it is unsuccessful in practice because the physics is over-simplified and hardware limitations are ignored, or regularity is not guaranteed, and unexpected hazardous motions can occur. This paper presents a reinforcement learning framework capable of learning ro-bust standing push recovery for bipedal robots that smoothly transfer to reality, providing only instantaneous proprioceptive observations. By combining original termination conditions and policy smoothness conditioning, we achieve stable learning, sim-to-real transfer and safety using a policy without memory nor explicit history. Reward engineering is then used to give insights into how to keep balance. We demonstrate its performance in reality on the lower-limb medical exoskeleton Atalante.

Social learning, copying other's behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others' learning as an external knowledge base.

This paper presents a new algorithm for distributed on-line evolutionary learning in swarm robotics. The challenge we address is to cope with the limited computation and communication capabilities of low cost robots, which are often used in swarm robotics. In order to do so, the algorithm decouples computation and communication and ensures learning of efficient control policies even when only a limited amount of information can be exchanged between neighbouring robots. We show experimentally that this algorithm is both remarkably robust with respect to its meta-parameter values, and able to adapt automatically to the available communication bandwidth.

Autonomous robots require online trajectory planning capability to operate in the real world. Efficient offline trajectory planning methods already exist, but are computationally demanding, preventing their use online. In this paper, we present a novel algorithm called Guided Trajectory Learning that learns a function approximation of solutions computed through trajectory optimization while ensuring accurate and reliable predictions. This function approximation is then used online to generate trajectories. This algorithm is designed to be easy to implement, and practical since it does not require massive computing power. It is readily applicable to any robotics systems and effortless to set up on real hardware since robust control strategies are usually already available. We demonstrate the computational performance of our algorithm on flat-foot walking with the self-balanced exoskeleton Atalante.

This paper deals with the programmability of a swarm of bio-micro-robots in order to display self-assembling behaviors into specific shapes. We consider robots that are DNA-functionalized micro-beads capable of sensing and expressing signals as well as self-assembling. We describe an in vitro experimentation with a million of micro-beads conditionally aggregating into clusters. Using a realistic simulation, we then address the question of how to automatically design the reaction networks that define the micro-robots' behavior, to self-assemble into a specific shape at a specific location. We use bioNEAT, an instantiation of the famous NEAT algorithm capable of handling chemical reaction networks, and CMA-ES to optimize the behavior of each micro-bead. As in swarm robotics, each micro-bead shares the same behavioral rules and the general outcome depends on interactions between neighbors and with the environment. Results obtained on four different target functions show that solutions optimized with evolutionary algorithms display efficient self-assembling behaviors, improving over pure hand-designed networks provided by an expert after a week-long trials and errors search. In addition, we show that evolved solutions are able to self-repair after damage, which is a critical property for smart materials.

We are delighted to introduce this Special Issue on the Evolution of Physical Systems, the culmination of a series of workshops organized around the topic that began with ALIFE XIII (2012) and ran through ECAL 2013, ALIFE XIV (2014), and ECAL 2015.Inspired by our mutual interests, we coined the term evolution of physical systems (EPS) to describe evolutionary approaches that occur in real-world physical substrates rather than in simulation. We deliberately chose this term to be broad enough to encompass both parallel embodied evolution [19], in which evolution is distributed across a population of robots, and more classical evolutionary robotics work, where evaluation is serialized on a single robot, as by Floreano and Mondada [5].The evolution of physical systems has its roots in the embodiment philosophy of Rodney Brooks, who famously said “the world is its own best model” [3]. Brooks' emphatic critique of the symbol system hypothesis argued that relatively simple systems, developed and grounded in suitably complex real-world environments, can lead to the emergence of complex behaviors. Contemporary approaches to the EPS aim at automating Brooks' vision by exploiting evolutionary algorithms on systems that “live” and “evolve” in the real world, that is, whose behavior (and possibly their form) are as grounded as possible in the environment.Embodied evolutionary algorithms first gained prominence with the “Sussex approach” of Harvey et al. [7] in the 1990s. The Sussex gantry-based robot [4, 6], driven by a neural network, was capable of robust behaviors in a noisy real-world environment. Their motivation for using the real-world rather than a simulated environment stemmed largely from the fact that in that era of limited computational power, it was faster and more efficient to test networks in situ than it was to develop and run a simulator that realistically modeled sensor noise. This notion, that it is sometimes more practical (and computationally efficient) to dispense with simulation entirely and evolve behaviors directly in the real world, persists at the heart of the EPS.Two other results of the Sussex group are particularly noteworthy. First is Adrian Thompson's silicon evolution of a field-programmable gate array (FPGA) [17, 18]. The chosen task was to evolve a circuit capable of discriminating between 1- and 10-kHz signals without the use of a clock. By evolving in silicon, rather than in simulation, Thompson was able to generate nearly inscrutable solutions that exploited the analogue (and non-simulable) nature of FPGAs. This idea of using embodiment to arrive at “novel surprise” solutions is a further guiding principle of the EPS. The second is Jakobi's essential work on the reality gap [8], which highlighted the difficulty in transferring results evolved in simulation into the real world. Jakobi notes that the best results emerge when “the noise levels of the simulation have similar amplitudes to those observed in reality,” and points out that these approaches become less feasible as environments and sensors become more complex.The practice of evolving outside of simulators grew from there. Floreano and Mondada [5] were able to evolve robust wall-following behavior in a tethered Khepera robot through embodiment. Watson et al. [19] took embodiment a step further by simultaneously embedding an entire population of evolving TupperBots into a shared arena—and demonstrated how the behaviors that emerged through this process were qualitatively different, and more effective, than those evolved in a simulated environment (their use of an electrified floor as an alternative to tethers was particularly innovative). Zykov et al. evolved dynamic open-loop gaits on a large hexapod robot, remarking upon the amount of labor required to reset the large robot between physical trials [21].Several of our own research areas have been in this realm of problems that are easier to test physically than they are to simulate realistically. In their work on “evolutionary fabrication” [16, 12], Rieffel and colleagues ran an embodied evolutionary algorithm on a 3D printer, learning to produce a variety of shapes out of viscous extruded materials such as silicone. These efforts led to novel shapes (and novel means of producing the shapes) that no human would have designed. Similarly, and more recently, Rieffel's work on the evolution of vibration-based gaits for tensegrity robots [15] relied purely on physical trials, and led not only to fast and dynamically complex gaits, but also to a state-machine-based controller for a tensegrity robot able to chase targets [9].While not using a simulator at all has many benefits, evolutionary robotics can take advantage of both simulators and real experiments to accelerate the evolutionary process while keeping a direct link with the real world. A first idea is to start evolution in simulation and then finish it on the real robot(s); this is, for instance, what Lipson and Pollack did to evolve the morphology and controllers of 3D-printed robots [13]. However, this approach means that designs for the real world are variants of those evolved in simulation, that is, robots are more grounded in the simulation than in the real world. A second, classic idea is to use data collected from tests on the real robot to improve the simulator, ideally to make the reality gap so small that it becomes unnoticeable [2, 20]. Nevertheless, to do so, simulators need to be generic enough to model almost anything that could happen, which, in turn, requires acquiring a considerable amount of data in the real world. More recently, a weaker version of this idea has emerged: Instead of trying to correct the simulator, one could use a few tests in the real world to learn to avoid the behaviors that exploit phenomena that are inaccurately simulated. Following this concept, Koos et al. managed to evolve controllers for six-legged robots and Khepera-like robots with only a dozen tests in the real world [10, 11, 14].Despite numerous advances in both robotics and technology, Rodney Brooks' prescient words about the crucial role of embodiment in robotics remain as true as ever, and the EPS is as exciting as ever.We the editors are delighted to present the articles that were selected to be published in this issue. Collectively they provide an inspiring snapshot of the state of the evolution of physical systems.Jacobsen and colleagues describe an experimental setup that removes the burden of manually setting the environment (e.g., moving the robot(s) and object(s) back to their initial positions and orientations) from the human supervisor. This is particularly relevant in evolutionary robotics, where a large number of trials are generally required, all starting from similar initial conditions. The authors present a proof of concept, tested with a real robotics setup in an arena of 1 by 0.5 m, which makes use of an industrial robot arm, a camera, and fiducial markers that can be stuck on top of robots and objects alike. These fiducial markers are used both for monitoring the experiments, by tracking positions and orientations, and as visual anchors for the robot arm to grab so as to move back robots and objects to their initial positions. Beyond evolutionary robotics, the proposed setup may also be used for other robotics setups involving multiple trials (e.g., reinforcement learning and embodied evolution), whether for resetting the experimental setup, as demonstrated in this article, or for other, more dynamical interventions (e.g., by conditionally moving objects around or transporting robots during the course of an experiment). To ensure that their system can be adopted and easily deployed by others, the authors have included the source code and a detailed description of the technical aspects of their setup.In their contribution, Scheper and de Croon describe how choosing an appropriate level of abstraction can improve the transfer of evolved solutions across the reality gap. The task domain they have have chosen is a decentralized formation flying task for a group of three small airborne vehicles controlled by neural networks. They compare the performance of a low-level controller that has precise control over vehicle dynamics (pitch rate, pitch angle, etc.) against a higher level controller that uses a more abstract velocity setpoint. After evolution (in simulation), the more abstract-level controller was better able to survive transfer into the real world. Scheper and de Croon suggest that the higher-level controller is better able to perform a kind of environmental exploration by inherently modeling the behavior of the other vehicles in the swarm. As they point out, an advantage of more abstracted controllers is that it reduces the necessity of high-fidelity simulation—an important advantage for evolutionary robotics tasks. We were impressed by the insights of this article in regard to the tradeoffs between simulator fidelity and real-world behavior. The flight videos on YouTube are also quite impressive.The work of Vujovic and colleagues is very much in the spirit of this special issue. They introduce a novel robot-building robot composed of a hot-melt extruder and an articulated arm, which acts as a sort of 3D printer, capable of fabricating the legs of simple modular robots. An evolutionary developmental approach is used to “grow” the designs of the legs, and once printed, the locomotive abilities of the printed robots are automatically evaluated in situ. This type of physically embodied integrated design-fabricate-test loop, in which one robot evolves and assembles other robots, is very much in the vein of Lipson and Pollack's GOLEM robots [13], moving the field one step closer to 3D-printed robots that can walk themselves off the printer.Preen and Bull present an interesting theoretical analysis followed up with an actual physical case study. Evaluation of physical artefacts is commonly a time-consuming aspect of the EPS, so a thorough analysis of methods to minimize the number of required evaluations can greatly boost applicability. Preen and Bull employ the abstract NKCS model of coevolution to achieve this: It enables informed sampling of candidate solutions in a coevolutionary scenario, where various components of a physical system interact in combination with a neural-net-based surrogate model. A successful case study to develop a heterogeneous set of vertical wind turbines for wind farms shows the applicability and relevance of their method to the evolution of designs for physical artefacts without relying on simulation.For obvious practical reasons, most of the work in the EPS has been focused on the evolution of controllers for existing robots, that is, for robots with a fixed morphology. Nevertheless, most of those who are interested in the field dream of a much more ambitious vision: robots whose morphology and brains can evolve on their own, much in the same ways as animals evolve.In their contribution, Jelisavcic et al. investigate how such a vision could be achieved with a population of robots and 3D printing technology. Using the RoboGen system of 3D-printed modules [1], they first designed and 3D-printed two robots. In a second step, they put these two robots in an arena so that they could exchange their genome to create the genotype of a third robot, which they also 3D-printed. This first step of an evolutionary experiment is an interesting stepping stone to identify the challenges of setting up experiments in which a population of robots would evolve autonomously.Overall, these articles provide valuable contributions to the field of the evolution of physical systems, demonstrating that many of the current challenges of the field are more technological than scientific: To design machines that evolve in the physical world, we first need machines that can self-reproduce or, at least, autonomous machines that can in turn help other machines to evolve. And until high-fidelity computationally efficient simulators are developed, the role of physical embodiment in evolutionary robotics remains essential.

In this article, we are intested in the evolution of specialisation among a single population of heterogeneous robotic agents in a cooperative foraging task. In particular, we want to compare (1) the emergence and (2) fixation of genotypic polymorphism under two different selection methods: elitist and fitness- proportionate. We show that, while the emergence of specialists is easy under an elitist selection, this method cannot maintain heterogeneous behaviours throughout the whole simulation. In comparison a fitness-proportionate algorithm proves to be inefficient in evolving any cooperative strategy but ensures the conservation of heterogeneity when it is present in the population. We then reveal through additional experiments two key factors for the evolution of heterogenous behaviours in our task: (1) protection of genotypic diversity and (2) efficient selection of partners. We finally demonstrate this assertion and, while our main problem remains unsolved, we provide directions on how it could be successfully approached.

Mutualistic cooperation often requires multiple individuals to behave in a coordinated fashion. Hence, while the evolutionary stability of mutualistic cooperation poses no particular theoretical difficulty, its evolutionary emergence faces a chicken and egg problem: an individual cannot benefit from cooperating unless other individuals already do so. Here, we use evolutionary robotic simulations to study the consequences of this problem for the evolution of cooperation. In contrast with standard game-theoretic results, we find that the transition from solitary to cooperative strategies is very unlikely, whether interacting individuals are genetically related (cooperation evolves in 20% of all simulations) or unrelated (only 3% of all simulations). We also observe that successful cooperation between individuals requires the evolution of a specific and rather complex behaviour. This behavioural complexity creates a large fitness valley between solitary and cooperative strategies, making the evolutionary transition difficult. These results reveal the need for research on biological mechanisms which may facilitate this transition.

Embodied evolutionary robotics is a particular flavour of evolutionary robotics, where the evolutionary optimization of behaviours is achieved in an on-line and distributed fashion (Watson et al., 2002). The question asked in this paper is: does population size play a role in the evolution of particular behaviours? We experimentaly demonstrate that varying the number of robots and the size of the environment can lead to very different outcomes in terms of evolved behaviours.

This paper investigates the properties required to evolve Artificial Neural Networks for distributed control in modular robotics, which typically involves non-linear dynamics and complex interactions in the sensori-motor space. We investigate the relation between macro-scale properties (such as modularity and regularity) and micro-scale properties in Neural Network controllers. We show how neurons capable of multiplicative-like arithmetic operations may increase the performance of controllers in several ways whenever challenging control problems with non-linear dynamics are involved. This paper provides evidence that performance and robustness of evolved controllers can be improved by a combination of carefully chosen micro- and macro-scale neural network properties.

Most works in multicellular artificial ontogeny solve the halting problem by arbitrarily limiting the number of iterations of the developmental process. Hence, the trajectory of the developing organism in the phenotypic space is only required to come close to an accurate solution during a very short developmental period. Because of the well-known opportunism of evolution, there is indeed no reason for the organism to remain close to a good solution in other situations: if the development is continued after the limiting bound; if the environment is perturbed by some noise during the development; if the development takes place in different physical conditions. In order to increase the robustness of the solution against such hazards, a new stopping criterion for the developmental process is proposed, based on the stability of some internal energy of the organism during its development. Such adaptive stopping criterion biases evolution toward solutions in which robustness is an intrinsic property. Experimental results on different “French flag” problems demonstrate that enforcing stable developmental process makes it possible to produce solutions that not only accurately approximate the target shape, but also demonstrate near-perfect self-healing properties, as well as excellent generalization capabilities.

The SYMBRION project stands at the crossroads of artificial life and evolutionary robotics: a swarm of real robots undergoes online evolution by exchanging information in a decentralized Evolutionary Robotics Scheme: the diffusion of each individual's genotype depends both on its ability to survive in an unknown environment as well as its ability to maximize mating opportunities during its lifetime, which suggests an implicit fitness. This paper presents early research and prospective ideas in the context of large-scale swarm robotics projects, focusing on the open-ended evolutionary approach in the SYMBRION project. One key issue of this work is to perform on-board evolution in a spatially distributed population of robots. A real-world experiment is also described which yields important considerations regarding open-ended evolution with real autonomous robots.

Interested in Evolutionary Robotics, this paper focuses on the acquisition and exploitation of memory skills. The targeted task is a well-studied benchmark problem, the Tolman maze, requiring in principle the robotic controller to feature some (limited) counting abilities. An elaborate experimental setting is used to enforce the controller generality and prevent opportunistic evolution from mimicking deliberative skills through smart reactive heuristics. The paper compares the prominent NEAT approach, achieving the non-parametric optimization of Neural Nets, with the evolutionary optimization of Echo State Networks, pertaining to the recent field of Reservoir Computing. While both search spaces offer a sufficient expressivity and enable the modelling of complex dynamic systems, the latter one is amenable to robust parametric, linear optimization with Covariance Matrix Adaptation-Evolution Strategies.

This paper introduces a continuous model for Multi-cellular Developmental Design. The cells are fixed on a 2D grid and exchange "chemicals" with their neighbors during the growth process. The quantity of chemicals that a cell produces, as well as the differentiation value of the cell in the phenotype, are controlled by a Neural Network (the genotype) that takes as inputs the chemicals produced by the neighboring cells at the previous time step. In the proposed model, the number of iterations of the growth process is not pre-determined, but emerges during evolution: only organisms for which the growth process stabilizes give a phenotype (the stable state), others are declared nonviable. The optimization of the controller is done using the NEAT algorithm, that optimizes both the topology and the weights of the Neural Networks. Though each cell only receives local information from its neighbors, the experimental results of the proposed approach on the 'flags' problems (the phenotype must match a given 2D pattern) are almost as good as those of a direct regression approach using the same model with global information. Moreover, the resulting multi-cellular organisms exhibit almost perfect self-healing characteristics.

This paper deals with the possible benefits of perceptual learning in artificial intelligence. On the one hand, perceptual learning is more and more studied in neurobiology and is now considered as an essential part of any living system. In fact, perceptual learning and cognitive learning are both necessary for learning and often depend on each other. On the other hand, many works in machine learning are concerned with "abstraction" in order to reduce the amount of complexity related to some learning tasks. In the abstraction framework, perceptual learning can be seen as a specific process that learns how to transform the data before the traditional learning task itself takes place. In this paper, we argue that biologically inspired perceptual learning mechanisms could be used to build efficient low-level abstraction operators that deal with real-world data. To illustrate this, we present an application where perceptual-learning-inspired metaoperators are used to perform an abstraction on an autonomous robot visual perception. The goal of this work is to enable the robot to learn how to identify objects it encounters in its environment.

Code bloat, the excessive increase of code size, is an important issue in Genetic Programming (GP). This paper proposes a theoretical analysis of code bloat in the framework of symbolic regression in GP, from the viewpoint of Statistical Learning Theory, a well grounded mathematical toolbox for Machine Learning. Two kinds of bloat must be distinguished in that context, depending whether the target function lies in the search space or not. Then, important mathematical results are proved using classical results from Statistical Learning. Namely, the Vapnik-Chervonenkis dimension of programs is computed, and further results from Statistical Learning allow to prove that a parsimonious fitness ensures Universal Consistency (the solution minimizing the empirical error does converge to the best possible error when the number of examples goes to infinity). However, it is proved that the standard method consisting in choosing a maximal program size depending on the number of examples might still result in programs of infinitely increasing size with their accuracy; a more complicated modification of the fitness is proposed that theoretically avoids unnecessary bloat while nevertheless preserving the Universal Consistency.