Credit card transactions predicted to be fraudulent by automated detection systems are typically handed over to human experts for verification. To limit costs, it is standard practice to select only the most suspicious transactions for investigation. We claim that a trade-off between exploration and exploitation is imperative to enable adaptation to changes in behavior (concept drift). Exploration consists of the selection and investigation of transactions with the purpose of improving predictive models, and exploitation consists of investigating transactions detected to be suspicious. Modeling the detection of fraudulent transactions as rewarding, we use an incremental Regression Tree learner to create clusters of transactions with similar expected rewards. This enables the use of a Contextual Multi-Armed Bandit (CMAB) algorithm to provide the exploration/exploitation trade-off. We introduce a novel variant of a CMAB algorithm that makes use of the structure of this tree, and use Semi-Supervised Learning to grow the tree using unlabeled data. The approach is evaluated on a real dataset and data generated by a simulator that adds concept drift by adapting the behavior of fraudsters to avoid detection. It outperforms frequently used offline models in terms of cumulative rewards, in particular in the presence of concept drift.

Usually in reinforcement learning, the goal of the agent is to maximize the expected return. However, in practical applications, algorithms that solely focus on maximizing the mean return could be inappropriate as they do not account for the variability of their solutions. Thereby, a variability measure could be included to accommodate for a risk-sensitive setting, i.e. where the system engineer can explicitly define the tolerated level of variance. Our approach is based on multi-objectivization where a standard single-objective environment is extended with one (or more) additional objectives. More precisely, we augment the standard feedback signal of an environment with an additional objective that defines the variance of the solution. We highlight that our algorithm, named risk-sensitive Pareto Q-learning, is (1) specifically tailored to learn a set of Pareto non-dominated policies that trade-off these two objectives. Additionally (2), the algorithm can also retrieve every policy that has been learned throughout the state-action space. This in contrast to standard risk-sensitive approaches where only a single trade-off between mean and variance is learned at a time.

This paper addresses one of the main challenges on the way to an offshore transnational multi-terminal dc (MTdc) network: its control and operation. The main objective is to demonstrate the feasibility of using reinforcement learning (RL) techniques to control, in real time, a multi-terminal dc network aimed at integrating offshore wind farms (OWFs). This method of controlling MTdc networks using RL techniques is called Online Distributed Voltage Control (ODVC). The ODVC strategy uses Continuous Action Reinforcement Learning Automata (CARLA) to optimize power flows in real time. To validate the effectiveness of the proposed control method, dynamic simulations are carried out using a MTdc grid model composed of six nodes, interconnecting three offshore wind farms to three European countries. The results obtained demonstrate the advantages of implementing an online distributed voltage control strategy to obtain feasible controlled power flows with low transmission losses. The results obtained demonstrate the feasibility of the proposed method to control, in real time, MTdc networks and that the RL techniques are well-suited for this problem due to their inherent advantages of coping with stochastic environments.

This extended abstract provides a brief overview of my PhD research on multi-objectivization and ensemble techniques in reinforcement learning.

Multi-objective problems with correlated objectives are a class of problems that deserve specific attention. In contrast to typical multi-objective problems, they do not require the identification of trade-offs between the objectives, as (near-) optimal solutions for any objective are (near-) optimal for every objective. Intelligently combining the feedback from these objectives, instead of only looking at a single one, can improve optimization. This class of problems is very relevant in reinforcement learning, as any single-objective reinforcement learning problem can be framed as such a multi-objective problem using multiple reward shaping functions. After discussing this problem class, we propose a solution technique for such reinforcement learning problems, called adaptive objective selection. This technique makes a temporal difference learner estimate the Q-function for each objective in parallel, and introduces a way of measuring confidence in these estimates. This confidence metric is then used to choose which objective's estimates to use for action selection. We show significant improvements in performance over other plausible techniques on two problem domains. Finally, we provide an intuitive analysis of the technique's decisions, yielding insights into the nature of the problems being solved.

Many interesting optimization problems cannot be solved efficiently. Recently, a lot of work has been done on meta-heuristic optimization methods that quickly find approximate solutions to otherwise intractable problems. While successful, the field suffers from a notable lack of reuse of methods, both in practical applications as in research. In this paper, we describe a semi-automated approach to design more re-usable methods, based on key principles of re-usability such as simplicity, modularity and generality. We illustrate this methodology by designing general metaheuristics (using hyperheuristics) and show that the methods obtained are competitive with the contestants of the Cross-Domain Heuristic Search Competition (2011). In particular, we find a method performing better than the competition's winner, which can be considered the state-of-the-art in domain-independent metaheuristic search.

Recent work on multi-objectivization has shown how a single-objective reinforcement learning problem can be turned into a multi-objective problem with correlated objectives, by providing multiple reward shaping functions. The information contained in these correlated objectives can be exploited to solve the base, single-objective problem faster and better, given techniques specifically aimed at handling such correlated objectives. In this paper, we identify ensemble techniques as a set of methods that is suitable to solve multi-objectivized reinforcement learning problems. We empirically demonstrate their use on the Pursuit domain.

In this work we propose learning an ensemble of policies related through potential-based shaping rewards via the off-policy Horde framework.

In this work we present a simple state-of-the-art selection hyperheuristic called Fair-Share Iterated Local Search (FS-ILS). FS-ILS is an iterated local search method using a conservative restart condition. Each iteration, a perturbation heuristic is selected proportionally to the acceptance rate of its previously proposed candidate solutions (after iterative improvement) by a domain-independent variant of the Metropolis condition. FS-ILS was developed in prior work using a semi-automated design approach. That work focused on how the method was found, rather than the method itself. As a result, it lacked a detailed explanation and analysis of the method, which will be the main contribution of this work. In our experiments we analyze FS-ILS's parameter sensitivity, accidental complexity and compare it to the contestants of the CHeSC (2011) competition.

Multi-objectivization is the process of transforming a single objective problem into a multi-objective problem. Research in evolutionary optimization has demonstrated that the addition of objectives that are correlated with the original objective can make the resulting problem easier to solve compared to the original single-objective problem. In this paper we investigate the multi-objectivization of reinforcement learning problems. We propose a novel method for the multi-objectivization of Markov Decision problems through the use of multiple reward shaping functions. Reward shaping is a technique to speed up reinforcement learning by including additional heuristic knowledge in the reward signal. The resulting composite reward signal is expected to be more informative during learning, leading the learner to identify good actions more quickly. Good reward shaping functions are by definition correlated with the target value function for the base reward signal, and we show in this paper that adding several correlated signals can help to solve the basic single objective problem faster and better. We prove that the total ordering of solutions, and by consequence the optimality of solutions, is preserved in this process, and empirically demonstrate the usefulness of this approach on two reinforcement learning tasks: a pathfinding problem and the Mario domain.

To solve multi-objective problems, multiple reward signals are often scalarized into a single value and further processed using established single-objective problem solving techniques. While the field of multi-objective optimization has made many advances in applying scalarization techniques to obtain good solution trade-offs, the utility of applying these techniques in the multi-objective multi-agent learning domain has not yet been thoroughly investigated. Agents learn the value of their decisions by linearly scalarizing their reward signals at the local level, while acceptable system wide behaviour results. However, the non-linear relationship between weighting parameters of the scalarization function and the learned policy makes the discovery of system wide trade-offs time consuming. Our first contribution is a thorough analysis of well known scalarization schemes within the multi-objective multi-agent reinforcement learning setup. The analysed approaches intelligently explore the weight-space in order to find a wider range of system trade-offs. In our second contribution, we propose a novel adaptive weight algorithm which interacts with the underlying local multi-objective solvers and allows for a better coverage of the Pareto front. Our third contribution is the experimental validation of our approach by learning bi-objective policies in self-organising smart camera networks. We note that our algorithm (i) explores the objective space faster on many problem instances, (ii) obtained solutions that exhibit a larger hypervolume, while (iii) acquiring a greater spread in the objective space.

Traffic jams and suboptimal traffic flows are ubiquitous in modern societies, and they create enormous economic losses each year. Delays at traffic lights alone account for roughly 10% of all delays in US traffic. As most traffic light scheduling systems currently in use are static, set up by human experts rather than being adaptive, the interest in machine learning approaches to this problem has increased in recent years. Reinforcement learning (RL) approaches are often used in these studies, as they require little pre-existing knowledge about traffic flows. Distributed constraint optimisation approaches (DCOP) have also been shown to be successful, but are limited to cases where the traffic flows are known. The distributed coordination of exploration and exploitation (DCEE) framework was recently proposed to introduce learning in the DCOP framework. In this paper, we present a study of DCEE and RL techniques in a complex simulator, illustrating the particular advantages of each, comparing them against standard isolated traffic actuated signals. We analyse how learning and coordination behave under different traffic conditions, and discuss the multi-objective nature of the problem. Finally we evaluate several alternative reward signals in the best performing approach, some of these taking advantage of the correlation between the problem-inherent objectives to improve performance.

In this work, we develop a new paradigm, called Meta-Evolutionary Algorithms, motivated by the challenging, continuous problems encountered in the domain of satisfiability in fuzzy logics (SAT∞). In Meta-Evolutionary Algorithms, the individuals in a population are optimization algorithms themselves. Mutation at the meta-population level is handled by performing an optimization step in each optimization algorithm, and recombination at the meta-population level is handled by exchanging information between different algorithms. We analyse different recombination operators and empirically show that simple Meta-Evolutionary Algorithms are able to outperform CMA-ES on a set of SAT∞benchmark problems.

Satisfiability in propositional logic is well researched and many approaches to checking and solving exist. In infinite-valued or fuzzy logics, however, there have only recently been attempts at developing methods for solving satisfiability. In this paper, we propose new benchmark problems and analyse the function landscape of different problem classes, focussing our analysis on plateaus. Based on this study, we develop Mixing CMA-ES (M-CMA-ES), an extension to CMA-ES that is well suited to solving problems with many large plateaus. We empirically show the relation between certain function landscape properties and M-CMA-ES performance.

Many industrial problems are inherently multi-objective, and require special attention to find different trade-off solutions. Typical multi-objective approaches calculate a scalarization of the different objectives and subsequently optimize the problem using a single-objective optimization method. Several scalarization techniques are known in the literature, and each has its own advantages and drawbacks. In this paper, we explore various of these scalarization techniques in the context of an industrial application, namely the engagement of a wet clutch using reinforcement learning. We analyse the approximate Pareto front obtainable by each technique, and discuss the causes of the differences observed. Finally, we show how a simple search algorithm can help explore the parameter space of the scalarization techniques, to efficiently identify possible trade-off solutions.

Evolutionary Strategies (ES) are a class of continuous optimization algorithms that have proven to perform very well on hard optimization problems. Whereas in earlier literature, both intermediate and discrete recombination operators were used, we now see that most ES, e.g. CMA-ES, use only intermediate recombination. While CMA-ES is considered state-of-the-art in continuous optimization, we believe that reintroducing discrete recombination can improve the algorithms' ability to escape local optima. Specifically, we look at using information on the problem's structure to create building blocks for recombination.