Multi-objective reinforcement learning (MORL) algorithms tackle sequential decision problems where agents may have different preferences over (possibly conflicting) reward functions. These algorithms often learn a set of policies, each optimized for a particular agent preference, that are later reused when optimizing policies for different preferences. We introduce a novel algorithm that builds upon Generalized Policy Improvement (GPI) to construct principled, formally-derived prioritization schemes that improve sample efficiency. These correspond to active-learning strategies by which the agent can identify (i) the most promising preferences/objectives to train on at each moment; and (ii) the most relevant previous experiences to learn policies for new agent preferences through a novel Dyna-style MORL method. We prove our algorithm is guaranteed to always converge to an optimal solution in a finite number of steps, or an $$\epsilon $$ -optimal solution (for a bounded $$\epsilon $$ ) if the agent can only identify sub-optimal policies. Our method monotonically improves the quality of its partial solutions while learning. We also introduce a bound that characterizes the maximum utility loss (with respect to the optimal solution) incurred by intermediate policies identified by our method during learning. Finally, we propose a novel epistemic uncertainty-aware extension of GPI that exploits high-confidence lower bounds to mitigate the impact of unreliable action-value estimates in GPI policies, and prove that it provides tighter performance bounds than the current state of the art. We empirically show that our method outperforms state-of-the-art MORL algorithms in challenging multi-objective tasks.

Multi-task reinforcement learning aims to quickly identify solutions for new tasks with minimal or no additional interaction with the environment. Generalized Policy Improvement (GPI) addresses this by combining a set of base policies to produce a new one that is at least as good—though not necessarily optimal—as any individual base policy. Optimality can be ensured, particularly in the linear-reward case, via techniques that compute a Convex Coverage Set (CCS). However, these are computationally expensive and do not scale to complex domains. The Option Keyboard (OK) improves upon GPI by producing policies that are at least as good—and often better. It achieves this through a learned meta-policy that dynamically combines base policies. However, its performance critically depends on the choice of base policies. This raises a key question: is there an optimal set of base policies—an optimal *behavior basis*—that enables zero-shot identification of optimal solutions for *any* linear tasks? We solve this open problem by introducing a novel method that efficiently constructs such an optimal behavior basis. We show that it significantly reduces the number of base policies needed to ensure optimality in new tasks. We also prove that it is strictly more expressive than a CCS, enabling particular classes of *non-linear* tasks to be solved optimally. We empirically evaluate our technique in challenging domains and show that it outperforms state-of-the-art approaches, increasingly so as task complexity increases.

We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies---each solving a particular task---and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce $h$-GPI, a multi-step extension of GPI that interpolates between these extremes---standard model-free GPI and fully model-based planning---as a function of a parameter, $h$, regulating the amount of time the agent has to reason. We prove that $h$-GPI's performance lower bound is strictly better than GPI's, and show that $h$-GPI generally outperforms GPI as $h$ increases. Furthermore, we prove that as $h$ increases, $h$-GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by $h$-GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate $h$-GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.

Multi-objective reinforcement learning algorithms (MORL) extend standard reinforcement learning (RL) to scenarios where agents must optimize multiple---potentially conflicting---objectives, each represented by a distinct reward function. To facilitate and accelerate research and benchmarking in multi-objective RL problems, we introduce a comprehensive collection of software libraries that includes: (i) MO-Gymnasium, an easy-to-use and flexible API enabling the rapid construction of novel MORL environments. It also includes more than 20 environments under this API. This allows researchers to effortlessly evaluate any algorithms on any existing domains; (ii) MORL-Baselines, a collection of reliable and efficient implementations of state-of-the-art MORL algorithms, designed to provide a solid foundation for advancing research. Notably, all algorithms are inherently compatible with MO-Gymnasium; and(iii) a thorough and robust set of benchmark results and comparisons of MORL-Baselines algorithms, tested across various challenging MO-Gymnasium environments. These benchmarks were constructed to serve as guidelines for the research community, underscoring the properties, advantages, and limitations of each particular state-of-the-art method.

In many real-world applications, reinforcement learning (RL) agents might have to solve multiple tasks, each one typically modeled via a reward function. If reward functions are expressed linearly, and the agent has previously learned a set of policies for different tasks, successor features (SFs) can be exploited to combine such policies and identify reasonable solutions for new problems. However, the identified solutions are not guaranteed to be optimal. We introduce a novel algorithm that addresses this limitation. It allows RL agents to combine existing policies and directly identify optimal policies for arbitrary new problems, without requiring any further interactions with the environment. We first show (under mild assumptions) that the transfer learning problem tackled by SFs is equivalent to the problem of learning to optimize multiple objectives in RL. We then introduce an SF-based extension of the Optimistic Linear Support algorithm to learn a set of policies whose SFs form a convex coverage set. We prove that policies in this set can be combined via generalized policy improvement to construct optimal behaviors for any new linearly-expressible tasks, without requiring any additional training samples. We empirically show that our method outperforms state-of-the-art competing algorithms both in discrete and continuous domains under value function approximation.

With fast increasing urbanization levels, adaptive traffic signal control methods have great potential for optimizing traffic jams. In particular, deep reinforcement learning (RL) approaches have been shown to be able to outperform classic control methods. However, deep RL algorithms are often employed as black boxes, which limits their use in the real-world as the decisions made by the agents can not be properly explained. In this paper, we compare different function approximations methods used to estimate de action-value function of RL-based traffic controllers. In particular, we compare (i) their expressiveness, based on the resulting performance of the learned policies, and (ii) their explainability capabilities. To explain the decisions of each method, we use Shapley Additive Explanations (SHAP) to show the impact of the agent's state features on each possible action. This allows us to explain the learned policies with a single image, enabling an understanding of how the agent behaves in the face of different traffic conditions. In addition, we discuss the application of post-hoc explainability models in the context of adaptive traffic signal control, noting their potential and pointing out some of their limitations. Comparing our resulting methods to state-of-the-art adaptive traffic signal controllers, we saw significant improvements in travel time, speed score, and throughput in two different scenarios based on real traffic data.

Improvement of traffic signal control (TSC) efficiency has been found to lead to improved urban transportation and enhanced quality of life. Recently, the use of reinforcement learning (RL) in various areas of TSC has gained significant traction; thus, we conducted a systematic literature review as a systematic, comprehensive, and reproducible review to dissect all the existing research that applied RL in the network-level TSC domain, called as RL in NTSC or RL-NTSC for brevity. The review only targeted the network-level articles that tested the proposed methods in networks with two or more intersections. This review covers 160 peer-reviewed articles from 30 countries published from 1994 to March 2020. The goal of this study is to provide the research community with statistical and conceptual knowledge, summarize existence evidence, characterize RL applications in NTSC domains, explore all applied methods and major first events in the defined scope, and identify areas for further research based on the explored research problems in current research. We analyzed the extracted data from the included articles in the following seven categories: (i) publication and authors’ data, (ii) method identification and analysis, (iii) environment attributes and traffic simulation, (iv) application domains of RL-NTSC, (v) major first events of RL-NTSC and authors’ key statements, (vi) code availability, and (vii) evaluation. This paper provides a comprehensive view of the past 26 years of research on applying RL to NTSC. It also reveals the role of advancing deep learning methods in the revival of the research area, the rise of using non-commercial microscopic traffic simulators, a lack of interaction between traffic and transportation engineering practitioners and researchers, and a lack of proposal and creation of testbeds which can likely bring different communities together around common goals.

Reinforcement learning is an efficient, widely used machine learning technique that performs well in problems with a reasonable number of states and actions. This is rarely the case regarding control-related problems, as for instance controlling traffic signals, where the state space can be very large. One way to deal with the curse of dimensionality is to use generalization techniques such as function approximation. In this paper, a linear function approximation is used by traffic signal agents in a network of signalized intersections. Specifically, a true online SARSA$(\lambda)$algorithm with Fourier basis functions (TOS($\lambda $)-FB) is employed. This method has the advantage of having convergence guarantees and error bounds, a drawback of non-linear function approximation. In order to evaluate TOS($\lambda $)-FB, we perform experiments in variations of an isolated intersection scenario and a scenario of the city of Cottbus, Germany, with 22 signalized intersections, implemented in MATSim. We compare our results not only to fixed-time controllers, but also to a state-of-the-art rule-based adaptive method, showing that TOS($\lambda $)-FB shows a performance that is highly superior to the fixed-time, while also being at least as efficient as the rule-based approach. For more than half of the intersections, our approach leads to less congestion and delay, without the need for the knowledge that underlies the rule-based approach.

Traffic congestion is a significant issue in urban areas and can cause adverse effects. In this paper, our proposal categorizes the traffic activity from video through three steps: vehicle monitoring, feature extraction, and classification. The vehicle monitoring step comprises an object detector based on a convolutional neural network and multi-object tracker. The feature extraction step uses information related to each detected vehicle, in various points of the road, to represent the traffic condition through three features: density, flow, and velocity. We tested on the UCSD dataset and achieved the best performance with 98.82% of accuracy, which outperformed the state-of-the-art methods.

Efficient urban mobility proved to be a major challenge for smart cities. A better use of the urban traffic infrastructure is key in the effort of mitigating the effects of traffic congestion. Popular approaches range from classical control and optimization to multiagent reinforcement learning (MARL). This paper follows the latter track. Challenges here are manifold. First, most of the literature assumes that either traffic lights learn (while drivers do not change their behaviors) or viceversa. Second, no matter what kind of class of learning agents is considered, their actions are highly coupled (thus making the learning task harder). Third, when both classes of agents co-learn, these learning tasks are of different nature (from the point of view of MARL). Finally, a microscopic modeling and simulation of the this problem is not trivial. Therefore, this paper not only proposes a co-learning approach in which agents act in a shared environment, but also argues that this task needs to be formulated asynchronously. Besides, driver agents are able to update the value of the available actions by receiving information from other drivers. Results show that the co-learning approach outperforms other policies in terms of average travel time of all vehicles. Moreover, when co-learning is used, a better distribution of the trips within the network is observed.

Data stream classification poses many challenges for the data mining community when the environment is non-stationary, among which adaptation to the concept drifts, i.e., changes in the underlying concepts, is a major one. Two main ways to develop adaptive approaches are ensemble methods and incremental algorithms. Ensemble methods play an important role due to its modularity, which provides a natural way of adapting to change. Incremental algorithms are faster and have better anti-noise capacity than ensemble algorithms, but have more restrictions on concept drifting data streams. Thus, it is a challenge to combine the flexibility and adaptation of an ensemble classifier in the presence of concept drift, with the simplicity of use found in a single classifier with incremental learning. With this motivation, this work proposes an incremental, online, and probabilistic algorithm for classification as an effort of tackling concept drifting in data streams. The algorithm is called Incremental Gaussian Mixture Network for Non-Stationary Environments (IGMN-NSE) and is an adaptation of the IGMN algorithm. The two main contributions of IGMN-NSE in relation to the IGMN are predictive power improvement for classification tasks and adaptation to achieve a good performance in non-stationary environments. Extensive experiments with both synthetic and real-world data demonstrate that the IGMN-NSE can track the changing environments very closely, regardless of the type of concept drift.

Defining a reward function that, when optimized, results in a rapid acquisition of an optimal policy, is one of the most challenging problems involved when deploying reinforcement learning algorithms. The existing works on the optimal reward problem (ORP) propose mechanisms to design reward functions but their application is limited to specific sub-classes of single or multi-agent reinforcement learning problems. Moreover, these methods identify which rewards should be given in which situation, but not which aspects of the state or environment should be used when defining the reward function. Those methods also do not directly model how quickly an optimal policy can be learned by optimizing a given candidate reward function. In this paper, we define the extended optimal reward problem (EORP) which: i) can identify both reward features and reward weights that compose the reward function; ii) is general enough to deal with single and multi-agent reinforcement learning problems; iii) is scalable to problems with large number of agents learning simultaneously; iv) incorporates a learning effort metric in the evaluation of reward functions allowing the discovery of reward functions that result in faster learning. Experimental results on gridworld-like and traffic assignment scenarios are used to evaluate the efficiency of our approach in designing effective reward functions.

The multi-armed bandit (MAB) problem is concerned with an agent choosing which arm of a slot machine to play in order to optimize its reward. A family of reinforcement learning algorithms exists to tackle this problem, including a few variants that consider more than one agent (thus, characterizing a repeated game) and non-stationary variants. In this paper, we seek to evaluate the performance of some of these MAB algorithms and compare them with Q-learning when applied to a non-stationary repeated game, where commuter agents face thetask of learning how to choose a route that minimizes their travel times.

To reduce the air pollution and improve the energy efficiency, many countries and cities (e.g., Singapore) are on the way of introducing electric vehicles (EVs) to replace the vehicles serving in current traffic system. Effective placement of charging stations is essential for the rapid development of EVs, because it is necessary for providing convenience for EVs and ensuring the efficiency of the traffic network. However, existing works mostly concentrate on the mileage anxiety from EV users but ignore their strategic and competitive charging behaviors. To capture the competitive and strategic charging behaviors of the EV users, we consider that an EV user’s charging cost, which is dependent on other EV users’ choices, consists of the travel cost to access the charging station and the queuing cost in charging stations. First, we formulate the Charging Station Placement Problem (CSPP) as a bilevel optimization problem. Then, by exploiting the equilibrium of the EV charging game, we convert the bilevel optimization problem to a single-level one, following which we analyze the properties of CSPP and propose an algorithm Optimizing eleCtric vEhicle chArging statioN (OCEAN) to compute the optimal allocation of charging stations. Due to OCEAN’s scalability issue, we furthermore present a heuristic algorithm OCEAN with Continuous variables to deal with large-scale real-world problems. Finally, we demonstrate and discuss the results of the extensive experiments we did. It is shown that our approach outperform baseline methods significantly.

Domain-specific languages are a means of speeding up and easing software system development. Agent-based simulations are simulation systems in which autonomous entities, namely agents, are developed in order to reproduce, analyze, or predict a phenomenon under study. Despite its potential, available alternatives for agent-based modeling and simulation benefit from recurrent domain-related concepts, such as domain concerns and standard strategies for creating and initializing entities, in a limited way. Such elements, when represented in models, can potentially ease the modeling and simulation. We propose a domain-specific modeling language that provides ready-to-use concepts for these recurrent agent-based simulation elements with a modularized notation.The language is targeted to the agent-based modeling and simulation domain and is currently focused on modeling the simulated environment. A user study provides evidence that our language decreases the time needed to understand agent-based simulations.

The traffic assignment problem (TAP) plays a key role in the context of efficient urban mobility. The TAP can be approached from various perspectives. One of the fundamental models to solve the TAP is the so-called User Equilibrium (UE), which assumes that drivers behave rationally aiming at minimising their travel costs. However, this is a complex optimisation problem. To this regard, in this paper we propose the use of the GRASP metaheuristic to provide approximate solutions to the UE. The path relinking mechanism is also used to increase the coverage of the solutions space. We advance the state-of-the-art by proposing a novel modelling for the TAP, through which one can adjust the granularity of the search space, thus making a more efficient, directed local search. We also devise an efficient assignment evaluation scheme that avoids redundant computations during the local search process. Additionally, we develop a novel greedy procedure for generating enhanced initial solutions for the GRASP algorithm. Based on experiments, we demonstrate that our approach outperforms classical algorithms, providing solutions that are significantly closer to the UE. Moreover, our empirical results show that the stability and fairness levels achieved by our approach are considerably better than those achieved by other methods.

Since in many cities transport infrastructure is operating at or beyond capacity, novel approaches to organize urban mobility are gaining attraction. However, assessing the benefits of a measure that has disruptive capacity in a complex system requires a carefully designed research. This paper takes a recent idea for urban mobility - flexible road trains - and illustrates the computational and research challenges of realizing its full potential and describing its social, ecological and economical impact.

An important stage in traffic planning is traffic assignment, which seeks to reproduce the way drivers select their routes. It assumes that each driver is aware of a number of routes to travel from an origin to a destination, that it performs some experimentation, and that it selects rationally the route with the highest utility. This is the basis for many approaches that, in an iterative way, vary the combination of route choices in order to find one that maximizes the utility. This perspective is therefore a centralized, aggregate one. In reality, though, drivers may perform en-route experimentation, i.e., they deviate from the originally planned route. Thus, in this paper, individual drivers are considered as active and autonomous agents, which, instead of having a central entity assigning complete trips to each agent, build these trips by experimentation during the actual trip. Agents learn their routes by deciding, at each node, how to continue their trips to each one's destination, in a way to minimize their travel times. Because the choice of one agent does impact several others, this is a non-cooperative multiagent learning problem (thus stochastic), which is known for being much more challenging than single agent reinforcement learning. To illustrate this approach, results from two non-trivial networks are presented, which have thousands of learning agents, clearly configuring a hard learning problem. Results are compared to iterative, centralized methods. It is concluded that an agent-based perspective yields choices that are more aligned with the real-world situation because (i) trips are computed by the agent itself (and not provided to the agent by any central entity), and (ii) it is not based on pre-computed paths (rather, it is built during the trip itself).

In this paper, we design a distributed mechanism that is able to create a social convention within a large convention space for multiagent systems (MAS) operating on various topologies. Specifically, we investigate a language coordination problem in which agents in a dynamic MAS construct a common lexicon in a decentralized fashion. Agent interactions are modeled using a language game where every agent repeatedly plays with its neighbors. Each agent stochastically updates its lexicons based on the utility values of the received lexicons from its immediate neighbors. We present a novel topology-aware utility computation mechanism and equip the agents with the ability to reorganize their neighborhood based on this utility estimate to expedite the convention formation process. Extensive simulation results indicate that our proposed mechanism is both effective (able to converge into a large majority convention state with more than 90\% agents sharing a high-quality lexicon) and efficient (faster) as compared to state-of-the-art approaches for social conventions in large convention spaces.

Solving the traffic assignment problem (TAP) is an important step towards an efficient usage of the traffic infrastructure. A fundamental assignment model is the so-called User Equilibrium (UE), which may turn into a complex optimisation problem. In this paper, we present the use of the GRASP metaheuristic to approximate the UE of the TAP. A path relinking mechanism is also employed to promote a higher coverage of the search space. Moreover, we propose a novel performance evaluation function, which measures the number of vehicles that have an incentive to deviate from the routes to which they were assigned. Through experiments, we show that our approach outperforms classical algorithms, providing solutions that are, on average, significantly closer to the UE. Furthermore, when compared to classical methods, the fairness achieved by our assignments is considerably better. These results indicate that our approach is efficient and robust, producing reasonably stable assignments.

Holonic multi-agent system (HMAS) offers a promising approach to model complex systems. HMAS is based on self-similar entities defined in a holarchical organization. Although some models and frameworks have been proposed for holonic systems, there is

The discovery of gene regulatory networks is a major goal in the field of bioinformatics due to their relevance, for instance, in the development of new drugs and medical treatments. The idea underneath this task is to recover gene interactions in a global and simple way, identifying the most significant connections and thereby generating a model to depict the mechanisms and dynamics of gene expression and regulation. In the present paper we tackle this challenge by applying a genetic algorithm to Boolean-based networks whose structures are inferred through the optimization of a Tsallis entropy function, which has been already successfully used in the inference of gene networks with other search schemes. Additionally, wisdom of crowds is applied to create a consensus network from the information contained within the last generation of the genetic algorithm. Results show that the proposed method is a promising approach and that the combination of criterion function based on Tsallis entropy with an heuristic search such as genetic algorithms yields networks up to 50% more accurate when compared to other Boolean-based approaches.

Learning in coordination games has been extensively studied in the game theory and multi-agent learning literature. Most of this work has considered a low number of agents and/or states (typically two agent, two action games). When the number of states and/or joint actions increases, standard approaches for multi-agent learning have difficulties coping with a high number of agents due to the combinatorial explosion in the number of joint actions and joint states. In real-world applications, this is a common setting though. This paper introduces a methodology for learning to coordinate in stochastic games with many agents. More specifically, we introduce a structure where some agents have knowledge about joint actions and how they have performed in the past. We empirically investigate this method for multi-agent learning in a typical stochastic game involving a high number of agents. Experimental results show that the additional information and structure is translated into earlier and higher levels of coordination and thus to higher payoffs.

Cellular automata models for traffic movement assume that vehicles are particles without routes. However, if one is interested in analysing microscopic properties, it is necessary to assign a route to each trip. This paper discusses the latest developments in the ITSUMO traffic simulator. These developments aim at modeling more sophisticated drivers’ behaviors such as en-route decision-making. They were tested in two scenarios, one being a real-world traffic network. We extensively discuss the effects of the use of various routing algorithms, as well as ration demand/capacity, control measures, network topologies, and re-planning strategies.

In our daily lives we often face binary decisions where we seek to take either the minority or the majority side. One of these binary decision scenarios is the El Farol Bar Problem, which has been used to study how agents achieve coordination. Previous works have shown that agents may reach appropriate levels of coordination, mostly by looking at their own individual strategies that consider the complete history of the bar attendance. No structure of the network of players has been explicitly considered. Here we use the formalism of random boolean networks to help agents to make decisions considering a network of other decision-makers. This is especially useful because random boolean networks allow the mapping of actions of K other agents (hence not based on complete history) to the decision-making of each single individual. Therefore a contribution of this work is the fact that we consider agents as participants of a social network. In the original proposition for this problem, strategies would change within time and eventually would lead agents to, collectively, decide on a efficient attendance, at each time step. Hence there was no explicit modeling of such a social network. Our results using random boolean networks show a similar pattern of convergence to an efficient attendance, provided agents do experimentation with the number of boolean functions, have a good update strategy, and a certain number of neighbors is considered.

The discovery of the structure of genetic regulatory networks is of great interest for biologists and geneticists due to its pivotal role in organisms' metabolism. In the present paper we aim to investigate the inference power of genetic regulatory networks modeled as random boolean networks without the use of any prior biological information. The solutions space is explored by means of genetic algorithms, whose main goal is to find a consistent network given the target data obtained from biological experiments. We show that this approach succeeds in reconstructing a model with satisfactory level of accuracy, representing an useful tool to guide biologist towards the most probable interactions between the target genes.

One of the well studied issues in multiagent systems is the action-selection and sequencing problem where a goal is decomposed in tasks that can be performed in different ways and/or by different agents. This problem has been tackled under different approaches. In particular, for open, dynamic environments agents must be able to adapt to the changing organizational goals, available resources, their relationships to another agents, and so on. This problem is a key one in multi-agent systems and relates to models of adaptation, such as those observed among social insects. This paper shows how mechanisms from Swarm Intelligence are used to solve the action-selection and sequencing problem in dynamically changing environments that can have large number of agents and tasks.

This paper addresses the problem of multiagent task allocation in extreme teams. An extreme team is composed by a large number of agents with overlapping functionality operating in dynamic environments with possible inter-task constraints. We present eXtreme-Ants, an approximate algorithm for task allocation in extreme teams. The algorithm is inspired by the division of labor in social insects and in the process of recruitment for cooperative transport observed in ant colonies. Division of labor offers fast and efficient decision-making, while the recruitment ensures the allocation of tasks that require simultaneous execution. We compare eXtreme-Ants with two other algorithms for task allocation in extreme teams and we show that it achieves balanced efficiency regarding quality of the solution, communication, and computational effort.

Traditional clustering methods have been usually developed in a centralized fashion. One reason for this is that this form of clustering relies on data structures that must be accessed and modified at each step of the clustering process. Another issue with classical clustering methods is that they need some hints about the target clustering. These hints include for example the number of clusters, the expected cluster size, or the minimum density of clusters. In this paper we propose a distributed clustering algorithm that is inspired by the organization of bee colonies. The performance of our algorithm in terms of the standard F-measure and rand index shows that it is possible to achieve results that are comparable to those from centralized approaches.

In this paper we introduce RL-CD, a method for solving reinforcement learning problems in non-stationary environments. The method is based on a mechanism for creating, updating and selecting one among several partial models of the environment. The partial models are incrementally built according to the system's capability of making predictions regarding a given sequence of observations. We propose, formalize and show the efficiency of this method both in a simple non-stationary environment and in a noisy scenario. We show that RL-CD performs better than two standard reinforcement learning algorithms and that it has advantages over methods specifically designed to cope with non-stationarity. Finally, we present known limitations of the method and future works.

This work reports on the ATUCG environment (Agent-based environmenT for aUtomatiC annotation of Genomes). It consists of three layers, each having several agents in charge of performing repetitive and time-consuming tasks. Layer I aims at automating the tasks behind the process of finding ORFs (Open Reading Frames). Layer II (the core of our approach) is associated with three main tasks: extraction and formatting of data, automatic annotation of data regarding profiles or families of proteins, and generation and validation of rules to automatically annotate the Keywords field in the SWISS-PROT database. Layer III permits the user to check the correctness of the automatic annotation. This environment is being designed having the sequencing of the Mycoplasma hyopneumoniae in mind. Thus examples are presented using data of organisms of the Mycoplasmataceae family. We have concentrated the developments in layer II because this is the most general one and because it focusses on machine learning algorithms, a characteristic which is not usual in annotation systems. Results regarding this layer show that with learning (individual or colaborative), agents are able to generate rules for annotation which achieve better results than those reported in the literature.