Deep reinforcement learning (DRL) algorithms have shown increasing promise for autonomous decision-making and continuous control, yet their practical value relative to established control methods remains difficult to assess under realistic control-engineering conditions. Existing evaluations are often narrow, application-specific, or based on inconsistent assumptions. As a result, it remains unclear when learned controllers are suitable alternatives to conventional designs. This paper presents a systematic assessment framework that compares four prominent DRL controllers with a classical control baseline across a diverse set of applied control problems, including non-minimum phase dynamics, flexible mechanical systems, nonlinear marine control, and aerial robotics. The framework standardises modelling assumptions, reward design, controller tuning, computational budgets, performance metrics, robustness tests, and deployment conditions. This allows fair and reproducible analysis across methods and applications. The evaluation covers tracking accuracy, settling time, overshoot, control effort, actuator saturation, disturbance rejection, model uncertainty, robustness margins, and sim-to-real transfer. The results show that performance depends strongly on the algorithmic structure, system dynamics, and operating conditions. Learned controllers provide advantages in some cases, particularly for nonlinearities, constraints, and uncertain operating regimes. Classical control remains competitive in accuracy, simplicity, and reliability for several benchmark conditions. Under combined saturation and unmodelled dynamics, the learned controllers maintained stable operation where the classical baseline became unstable. Real-time deployment on a quadrotor platform further demonstrated stable sim-to-real transfer. The study establishes a rigorous and reproducible evidence base for assessing the practical suitability of these data-driven control algorithms in applied control. It clarifies the trade-offs between learning-based and conventional control. This addresses a critical gap that cannot be easily inferred from isolated, system-specific studies.

The reward function is an essential component in robot learning. Reward directly affects the sample and computational complexity of learning, and the quality of a solution. The design of informative rewards requires domain knowledge, which is not always available. We use the properties of the dynamics to produce system-appropriate reward without adding external assumptions. Specifically, we explore an approach to utilize the Lyapunov exponents of the system dynamics to generate a system-immanent reward. We demonstrate that the Sum of the Positive Lyapunov Exponents (SuPLE) is a strong candidate for the design of such a reward. We develop a computational framework for the derivation of this reward, and demonstrate its effectiveness on classical benchmarks for sample-based stabilization of various dynamical systems. It eliminates the need to start the training trajectories at arbitrary states, also known as auxiliary exploration. While the latter is a common practice in simulated robot learning, it is unpractical to consider to use it in real robotic systems, since they typically start from natural rest states such as a pendulum at the bottom, a robot on the ground, etc. and can not be easily initialized at arbitrary states. Comparing the performance of SuPLE to commonly-used reward functions, we observe that the latter fail to find a solution without auxiliary exploration, even for the task of swinging up the double pendulum and keeping it stable at the upright position, a prototypical scenario for multi-linked robots. SuPLE-induced rewards for robot learning offer a novel route for effective robot learning in typical as opposed to highly specialized or fine-tuned scenarios. Our code is publicly available for reproducibility and further research.

The mastery of skills, such as balancing an inverted pendulum, implies a very accurate control of movements to achieve the task goals. Traditional accounts of skilled action control that focus on either routinization or perceptual control make opposite predictions about the ways we achieve mastery. The notion of routinization emphasizes the decrease of the variance of our actions, whereas the notion of perceptual control emphasizes the decrease of the variance of the states we visit, but not of the actions we execute. Here, we studied how participants managed control tasks of varying levels of difficulty, which consisted of controlling inverted pendulums of different lengths. We used information-theoretic measures to compare the predictions of alternative accounts that focus on routinization and perceptual control, respectively. Our results indicate that the successful performance of the control task strongly correlates with the decrease of state variability and the increase of action variability. As postulated by perceptual control theory, the mastery of skilled pendulum control consists in achieving stable control of goals by flexible means.

One of the fundamental modes of learning in children is through curiosity. Children (and adults) interact with new people, learn about novel objects, activities and other stimuli through curiosity and other intrinsic motivations. Creating autonomous robots that learn continually through intrinsic curiosity may result in breakthroughs in artificial intelligence. Such robots could continue to learn about themselves and the world around them through curiosity, thus improving their abilities over their ‘lifetime’. Although recent works on curiosity in different fields have produced significant results, most of these works have focused on constrained simulated environments which do not involve human interaction. However, in real-world applications such as healthcare, home-assistance etc., robots generally have to interact with humans on a regular basis. In these scenarios, it is imperative that curiosity is directed towards seeking out and learning important information from the humans when needed rather than simply learning in an unsupervised manner. Further, there is limited work on how humans perceive such curious robots and whether humans prefer curious robots that adapt over time to other robots that simply perform their assigned tasks. In this workshop, our goal is to bring together researchers and practitioners in different multidisciplinary fields to discuss the role of robot curiosity in real-world applications and its implications in human-robot interaction (HRI).

One difficulty in using artificial agents for human-assistive applications lies in the challenge of accurately assisting with a person's goal(s). Existing methods tend to rely on inferring the human's goal, which is challenging when there are many potential goals or when the set of candidate goals is difficult to identify. We propose a new paradigm for assistance by instead increasing the human's ability to control their environment, and formalize this approach by augmenting reinforcement learning with human empowerment. This task-agnostic objective increases the person's autonomy and ability to achieve any eventual state. We test our approach against assistance based on goal inference, highlighting scenarios where our method overcomes failure modes stemming from goal ambiguity or misspecification. As existing methods for estimating empowerment in continuous domains are computationally hard, precluding its use in real time learned assistance, we also propose an efficient empowerment-inspired proxy metric. Using this, we are able to successfully demonstrate our method in a shared autonomy user study for a challenging simulated teleoperation task with human-in-the-loop training.

Being able to measure time, whether directly or indirectly, is a significant advantage for an organism. It allows for timely reaction to regular or predicted events, reducing the pressure for fast processing of sensory input. Thus, clocks are ubiquitous in biology. In the present article, we consider minimal abstract pure clocks in different configurations and investigate their characteristic dynamics. We are especially interested in optimally time-resolving clocks. Among these, we find fundamentally diametral clock characteristics, such as oscillatory behavior for purely local time measurement or decay-based clocks measuring time periods on a scale global to the problem. We include also sets of independent clocks ( clock bags), sequential cascades of clocks, and composite clocks with controlled dependence. Clock cascades show a condensation effect, and the composite clock shows various regimes of markedly different dynamics.

Polani, D., Biehl, M., & Ikegami, T., 'Towards information based space-time patterns as a foundation for agent representation in dynamical systems', paper presented at the 15th International Conference on the Synthesis and Simulation of Living Systems, Cancun, Mexico, 4-8 July 2016. eBook of Conference Proceedings, published by MIT Press, is available on line https://mitpress.mit.edu/books/proceedings-artificial-life-conference-2016

We consider the problem of the evolution of a code within a structured population of agents. The agents try to maximize their information about their environment by acquiring information from the outputs of other agents in the population. A naive use of information-theoretic methods would assume that every agent knows how to interpret the information offered by other agents. However, this assumes that it knows which other agents it observes, and thus which code they use. In our model, however, we wish to preclude that: It is not clear which other agents an agent is observing, and the resulting usable information is therefore influenced by the universality of the code used and by which agents an agent is listening to. We further investigate whether an agent that does not directly perceive the environment can distinguish states by observing other agents' outputs. For this purpose, we consider a population of different types of agents talking about different concepts, and try to extract new ones by considering their outputs only.

The number of mechanical degrees of freedom (DoFs) within rehabilitation robots directly influences the scope of the movements that a subject can perform when training walking. Currently, gait rehabilitation robots have a limited number of mechanical DoFs, as a consequence this limits the movements these robots can make possible. In this paper, the novel gait rehabilitation system CORBYS is presented which consists of the mobile platform and a powered orthosis which is attached to the platform. The CORBYS powered orthosis has 16 DoFs enabling more physiological movements, making it a state-of-the-art gait rehabilitation robotic system. With the sufficient number of DoFs to enable natural-like walking, the CORBYS robotic system enables the integration of the “learning gait by therapist demonstration” paradigm. This paper presents the fully integrated functional CORBYS gait rehabilitation system, with the focus on the implementation aspects which enable generation of the reference gait trajectory through learning by therapist demonstration, and the use of the generated trajectory in the robotic therapy session. The results of the initial evaluation of the robotic system obtained in tests with a selected patient are given in the paper.

We are interested in designing artificial universes for artificial agents. We view artificial agents as networks of highlevel processes on top of of a low-level detailed-description system. We require that the high-level processes have some intrinsic explanatory power and we introduce an extension of informational closure namely interaction closure to capture this. Then we derive a method to design artificial universes in the form of finite Markov chains which exhibit high-level processes that satisfy the property of interaction closure. We also investigate control or information transfer which we see as an building block for networks representing artificial agents

Best Student Paper Award. Attribution-NonCommercial-NoDerivs 3.0 United States

Integrating information gained by observing others via Social Bayesian Learning can be beneficial for an agent’s performance, but can also enable population wide information cascades that perpetuate false beliefs through the agent population. We show how agents can influence the observation network by changing their probability of observing others, and demonstrate the existence of a population-wide equilibrium, where the advantages and disadvantages of the Social Bayesian update are balanced. We also use the formalism of relevant information to illustrate how negative information cascades are characterized by processing increasing amounts of non-relevant information

The objective of the CORBYS project is to design and implement a robot control architecture that allows the integration of high-level cognitive control modules, such as a semantically-driven self-awareness module and a cognitive framework for anticipation of, and synergy with, human behaviour based on biologically-inspired information-theoretic principles. CORBYS aims to provide a generic control architecture to benefit a wide range of applications where robots work in synergy with humans, ranging from mobile robots such as robotic followers to gait rehabilitation robots. The behaviour of the two demonstrators, used for validating this architecture, will each be driven by a combination of task specific algorithms and generic cognitive algorithms. In this paper we focus on the generic algorithms based on information theory.

In this paper, we use empowerment, a recently introduced biologically inspired measure, to allow an AI player to assign utility values to potential future states within a previously unencountered game without requiring explicit specification of goal states. We further introduce strategic affinity, a method of grouping action sequences together to form “strategies,” by examining the overlap in the sets of potential future states following each such action sequence. We also demonstrate an information-theoretic method of predicting future utility. Combining these methods, we extend empowerment to soft-horizon empowerment which enables the player to select a repertoire of action sequences that aim to maintain anticipated utility. We show how this method provides a proto-heuristic for nonterminal states prior to specifying concrete game goals, and propose it as a principled candidate model for “intuitive” strategy selection, in line with other recent work on “self-motivated agent behavior.” We demonstrate that the technique, despite being generically defined independently of scenario, performs quite well in relatively disparate scenarios, such as a Sokoban-inspired box-pushing scenario and in a Pac-Man-inspired predator game, suggesting novel and principle-based candidate routes toward more general game-playing algorithms.

It has been hypothesized that the evolution of sensors is a pivotal driver for the evolution of organisms, and especially, as a crucial part of the perception-action loop, a driver for cognitive development.The questions of why and how this is the case are important: what are the principles that push the evolution of sensorimotor systems?An interesting aspect of this problem is the co-option of sensors for functions other than those originally driving their development (e.g. the auditive sense of bats being employed as a 'visual' modality).Even more striking is the phenomenon found in nature of sensors being driven to the limits of precision, while starting from much simpler beginnings.While a large potential for diversification and exaptation is visible in the observed phenotypes, gaining a deeper understanding of why and how this can be achieved is a significant problem.In this present paper, we will introduce a formal and generic informationtheoretic model for understanding potential drives of sensor evolution, both in terms of improving sensory ability and in terms of extending and/or shifting sensory function.

In reinforcement learning problems, the construction of subgoals has been identified as an important step to speed up learning and to enable skill transfer. For this purpose, one typically extracts states from various saliency properties of an MDP transition graph, most notably bottleneck states. Here we introduce an alternative approach to this problem: assuming a family of MDPs with multiple goals but with a fixed transition graph, we introduce the relevant goal information as the amount of Shannon information that the agent needs to maintain about the current goal at a given state to select the appropriate action. We show that there are distinct transition states in the MDP at which new relevant goal information has to be considered for selecting the next action. We argue that these transition states can be interpreted as subgoals for the current task class, and we use these states to automatically create a hierarchical policy, according to the well-established Options model for hierarchical reinforcement learning.

Based on the fact that information processing is costly, we study in this paper the trade-off between performance and informational requirements. Most importantly, we are interested in how local decisions can alleviate future cognitive burden, measured by the amount of sensory information an agent processes, without conceding performance. We introduce lookahead information as a novel concept to capture the long-term informational requirements and present an iterative method to determine the value of this quantity. Using an example problem, we show how these long-term considerations enable an agent to predict future effects of its actions on its informational burden, and to shape the course of the world to achieve more informationally parsimonious behaviour.

Shannon's information theory can be used to quantify morphological and topological features of a collective of agents in an arbitrary environment. In particular the ability of individual agents to extract information locally about global features of the collective can be quantified. Here, we considered chains of agents in a grid world. The agents are equipped with local sensors. We then quantified the amount of information the sensors contain about certain features global to the chain. Furthermore, we compared the amount of locally available information to the amount of information the whole collective could in principle acquire about a feature in different contexts.

Living organisms are under permanent pressure to take decisions with an impact on their success. Such decisions require information, which can be formulated in the precise sense of Shannon information. Since information processing is costly for organisms, this creates an adaptive pressure for cognition to be as informationally parsimonious as possible. Combining information theory with the theory of reinforcement learning for modeling tasks, we present a number of quantitative analyses how the cognitive burden of an agent deriving from a task can be relieved by the environment and, more specifically, its embodiment, i.e. how the agent “controller” is linked to the environment, via perception (in principle, but not further considered here) and action (this paper's main focus). The methodology presented offers a path towards a formal and quantitative treatment of Paul's and Pfeifer's concept of morphological computation in particular and their envisaged larger picture of offloading of computation onto the environment dynamics in general. In particular, it offers additional evidence for the central importance of the embodiment for the success of cognition.

The main target of this work is to improve human-robot interaction capabilities, by adding a new modality of sense, touch, to KASPAR, a humanoid robot. Large scale distributed skin-like sensors are designed and integrated on the robot, covering KASPAR at various locations. One of the challenges is to classify different types of touch. Unlike digital images represented by grids of pixels, the geometrical structure of the sensor array limits the capability of straightforward application of well-established approaches for image patterns. This paper introduces a novel histogram-based classification algorithm, transforming tactile data into histograms of local features termed as codebook. Tactile pattern can be invariant at periodical locations, allowing tactile pattern classification using a smaller number of training data, instead of using training data from everywhere on the large scale skin sensors. To generate the codebook, this method uses a two-layer approach, namely local neighbourhood structures and encodings of pressure distribution of the local neighbourhood. Classification is performed based on the constructed features using Support Vector Machine (SVM) with the intersection kernel. Real experimental data are used for experiment to classify different patterns and have shown promising accuracy. To evaluate the performance, it is also compared with the SVM using the Radial Basis Function (RBF) kernel and results are discussed from both aspects of accuracy and the location invariance property.

October 01 2010 Information Theory and Evolution. John Avery. (2003, World Scientific.) £33 (paper). 217 pages. In Special Collection: CogNet Daniel Polani Daniel Polani University of Hertfordshire Search for other works by this author on: This Site Google Scholar Author and Article Information Daniel Polani University of Hertfordshire *Adaptive Systems and Algorithms Research Groups, School of Computer Science, University of Hertfordshire, Hatfield, Hertfordshire AL10 9AB, UK. E-mail: [email protected] Online Issn: 1530-9185 Print Issn: 1064-5462 © 2010 Massachusetts Institute of Technology2010MIT Press Artificial Life (2010) 16 (4): 333–335. https://doi.org/10.1162/artl_r_00009 Cite Icon Cite Permissions Share Icon Share Facebook Twitter LinkedIn MailTo Views Icon Views Article contents Figures & tables Video Audio Supplementary Data Peer Review Search Site Citation Daniel Polani; Information Theory and Evolution. John Avery. (2003, World Scientific.) £33 (paper). 217 pages.. Artif Life 2010; 16 (4): 333–335. doi: https://doi.org/10.1162/artl_r_00009 Download citation file: Ris (Zotero) Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentAll JournalsArtificial Life Search Advanced Search This content is only available as a PDF. © 2010 Massachusetts Institute of Technology2010MIT Press Article PDF first page preview Close Modal You do not currently have access to this content.

In scenarios such as therapeutic modeling or pest control, one aims to suppress infective agents or maximize crop yields while minimizing the side-effects of interventions, such as cost, environmental impact, and toxicity. Here, we consider the eradication of persistent microbes (e.g., Escherichia coli, multiply resistant Staphylococcus aureus (MRSA-“superbug”), Mycobacterium tuberculosis, Pseudomonas aeruginosa) through medication. Such microbe populations consist of metabolically active and metabolically inactive (persistent) subpopulations. It turns out that, for efficient medication strategies, the two goals, eradication of active bacteria on one hand and eradication of inactive bacteria on the other, are in conflict. Using multiobjective optimization, we obtain a survey of the full spectrum of best solutions. We find that, if treatment time is limited and the total medication dose is constant, the application of the medication should be concentrated both at the beginning and end of the treatment. If the treatment time is increased, the medication should become increasingly spread out over the treatment period until it is uniformly spread over the entire period. The transition between short and long overall treatment times sees optimal medication strategies clustered into groups.

This paper presents an application of the Cascaded Information Bottleneck Method for real-time recognition of Human-Robot Interaction styles in robot-assisted play. This method, that we have developed, is implemented here for an adaptive robot that can recognize and adapt to children's play styles in real time. The robot rewards well-balanced interaction styles and encourages children to engage in the interaction. The potential impact of such an adaptive robot in robot-assisted play for children with autism is evaluated through a study conducted with seven children with autism in a school. A statistical analysis of the results shows the positive impact of such an adaptive robot on the children's play styles and on their engagement in the interaction with the robot.

Empowerment quantifies the choice available to an agent as the actuation channel capacity. However, not all such choices are sustainable: After some choices, the agent may not be able to return to its original state, or returning there may be costly. In this paper we explore whether empowerment can be adapted to obtain a measure of sustainability. As a straightforward modification, the agent's options is restricted to actions that are reversible within a given time horizon. We furthermore investigate the lengths of return paths and discuss their potential to indicate sustainability.

This paper presents a proof-of-concept of a robot that is adapting its behaviour on-line, during interactions with a human according to detected play styles. The study is part of the AuRoRa project which investigates how robots may be used to help children with autism overcome some of their impairments in social interactions. The paper motivates why adaptation is a very desirable feature of autonomous robots in human-robot interaction scenarios in general, and in autism therapy in particular. Two different play styles namely 'strong' and 'gentle', which refer to the user, are investigated experimentally. The model relies on Self-Organizing Maps, used as a classifier, and on Fast Fourier Transform to preprocess the sensor data. First experiments were carried out which discuss the performance of the model. Related work on adaptation in socially assistive and therapeutic work are surveyed. In future work, with typically developing and autistic children, the concrete choice of the robot's behaviours will be tailored towards the children's interests and abilities.

We explore how information theoretic quantities such as potential information flow (empowerment) can be used as a drive toward complex collective behaviour in the context of multi-agent systems. In a first experiment, we investigate the empowerment of two agents interacting in a grid world. We show that some conditions lead to higher empowerment than others, depending on the amount of interaction and the amount of information shared by the agents. We then investigate more deeply the tradeoff between freedom of the agents and the constraints they impose on each other. We show that there exist a trade-off between these where empowerment is maximized. In a third experiment, we show that agents behaving so as to maximize potential information transfer over time generate a wide range of complex collective behaviours. We then discuss how these notions can be compared to what happens in natural systems

Sensor evolution in nature aims at improving the acquisition of information from the environment and is intimately related with selection pressure toward adaptivity and robustness. Our work in the area indicates that information theory can be applied to the perception-action loop. This letter studies the perception-action loop of agents, which is modeled as a causal Bayesian network. Finite state automata are evolved as agent controllers in a simple virtual world to maximize information flow through the perception-action loop. The information flow maximization organizes the agent's behavior as well as its information processing. To gain more insight into the results, the evolved implicit representations of space and time are analyzed in an information-theoretic manner, which paves the way toward a principled and general understanding of the mechanisms guiding the evolution of sensors in nature and provides insights into the design of mechanisms for artificial sensor evolution.

This article describes a developmental system based on information theory implemented on a real robot that learns a model of its own sensory and actuator apparatus. There is no innate knowledge regarding the modalities or representation of the sensory input and the actuators, and the system relies on generic properties of the robot’s world, such as piecewise smooth effects of movement on sensory changes. The robot develops the model of its sensorimotor system by first performing random movements to create an informational map of the sensors. Using this map, the robot then learns what effects the different possible actions have on the sensors. After this developmental process, the robot can perform basic visually guided movement.

The classical approach to using utility functions suffers from the drawback of having to design and tweak the functions on a case by case basis. Inspired by examples from the animal kingdom, social sciences and games we propose empowerment, a rather universal function, defined as the information-theoretic capacity of an agent's actuation channel. The concept applies to any sensorimotor apparatus. Empowerment as a measure reflects the properties of the apparatus as long as they are observable due to the coupling of sensors and actuators via the environment. Using two simple experiments we also demonstrate how empowerment influences sensor-actuator evolution

The potential for animals to inflict damage on one another whilst competing for indivisible resources is a factor of crucial importance when determining pay-offs to such animals and consequent likelihood of adopting an aggressive resource procurement strategy, a less costly display-based alternative or just a retreat response. Using computer simulations of evolving agents, we assessed the effects of degree of damage potential on social aggression and resource procuring strategies. Furthermore, we assessed the effects of evolving ritualized displays used in contests over resources relative to damage potential. Our results showed that aggressive interactions increased in frequency when the degree of damage potential was low. Ritualized displays tended to reduce aggressive approaches and mortality rate where damage potential was high and low but this was not the case at the intermediate level where aggression and mortality rate increased.

April 01 2001 Guest Editors' Introduction: Special Issue on Sensor Evolution Kerstin Dautenhahn, Kerstin Dautenhahn University of Hertfordshire, UK Search for other works by this author on: This Site Google Scholar Daniel Polani, Daniel Polani Medical University Lübeck, Germany Search for other works by this author on: This Site Google Scholar Thomas Uthmann Thomas Uthmann University of Mainz, Germany Search for other works by this author on: This Site Google Scholar Author and Article Information Kerstin Dautenhahn University of Hertfordshire, UK Daniel Polani Medical University Lübeck, Germany Thomas Uthmann University of Mainz, Germany Online Issn: 1530-9185 Print Issn: 1064-5462 © 2001 Massachusetts Institute of Technology2001 Artificial Life (2001) 7 (2): 95–97. https://doi.org/10.1162/106454601753138952 Cite Icon Cite Permissions Share Icon Share Facebook Twitter LinkedIn MailTo Views Icon Views Article contents Figures & tables Video Audio Supplementary Data Peer Review Search Site Citation Kerstin Dautenhahn, Daniel Polani, Thomas Uthmann; Guest Editors' Introduction: Special Issue on Sensor Evolution. Artif Life 2001; 7 (2): 95–97. doi: https://doi.org/10.1162/106454601753138952 Download citation file: Ris (Zotero) Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentAll JournalsArtificial Life Search Advanced Search This content is only available as a PDF. © 2001 Massachusetts Institute of Technology2001 Article PDF first page preview Close Modal You do not currently have access to this content.

In this article we study a model for the evolution of the spectral sensitivity of visual receptors for agents in a continuous virtual environment. The model uses a genetic algorithm (GA) to evolve the agent sensors along with the control of the agents by requiring the agents to solve certain tasks in the simulation environment. The properties of the evolved sensors are analyzed for different scenarios. In particular, it is shown that the GA is able to find a balance between sensor costs and agent performance in such a way that the spectral sensor sensitivity reflects the emission spectrum of the target objects and that the capability of the sensors to evolve can help the agents significantly in adapting to their task.