MNK landscapes are multi-objective combinatorial optimization problems. For MNK landscapes, computing the entire set of Pareto local optima and the Pareto front by full enumeration quickly becomes infeasible within a reasonable amount of time. Approximation methods are faster, but cannot guarantee that all Pareto local optima or Pareto non-dominated solutions are found. To address this, we propose a multi-objective gray-box optimization algorithm based on hyperplane elimination. By exploiting gray-box information, the number of evaluations is reduced by maintaining and updating changes in subfunction values of the objectives instead of re-evaluating the entire bitstring after each bit flip. In addition, hyperplanes of bitstrings can be eliminated from the search space by knowing all dependencies between the bits and the delta values. Since this algorithm identifies all Pareto local optima, the Pareto front can be obtained with very low additional cost. We provide theoretical proofs of correctness and experimental results on adjacent, random and p-correlated MNK landscapes showing speed-ups of several orders of magnitude. Further improvements on runtime are obtained by using a bit reordering heuristic and a prefix mechanism. Altogether, our proposed method enables the computation of exact Pareto sets of MNK landscapes even for large bit lengths.

In this paper, we address the problem of efficiently computing finite-size approximations of the set of -locally optimal solutions of a given multi-objective optimization problem (MOP). Such sets are in particular interesting in the context of multi-objective multimodal optimization (MMO). To this end, we first propose a bounded archiver, ArchiveUpdateLQ,∈B, that is, a modification of a previously proposed unbounded archiver. These archivers can be used as external archivers to in principle any multi-objective evolutionary algorithm (MOEA). In order to reduce the computational cost compared to such archive equipped MOEAs, we propose, in a next step LQ,∈MOEA. This evolutionary algorithm directly uses ArchiveUpdateLQ,∈B for the selection process and hence does not need an external archive for the computation of ∈-locally optimal solutions. We further propose a hybrid of LQ,∈MOEA with a multi-objective continuation method, which significantly improves the accuracy of the obtained solutions in case the gradient information is at hand. Finally, we show some numerical results that demonstrate the benefit of both the bounded archiver and the new MOEAs.

In many real-world problems, instances arrive in a stream which is likely to experience drift in the instance space over time. If a classical algorithm selector is trained offline, i.e., on an initial part of the instance stream, downstream performance is often negatively impacted due to drift in the instance data. To overcome this limitation of classical algorithm selectors, we propose a novel online automated algorithm selection framework that first uses instance features to detect drift, and then periodically retrains a selector if drift occurs, ensuring continuity of performance in face of data-drift. To further improve both the effectiveness and efficiency of retraining, we also propose a process to continuously gather new training samples on the fly. Empirical comparison using a bin-packing scenario under three different drift scenarios shows that our framework is efficient in terms of the computational effort required to train a selector while maintaining good performance with respect to accuracy compared to several baselines.

MNK-landscapes are a class of multi-objective combinatorial optimisation problems that simulate interactions between system components with adjustable parameters. Recently, heterogeneous MNK-landscapes were introduced, which feature objectives with varying interdependencies, offering a new direction in multi-objective (multi-modal) landscape research. This study benchmarks various evolutionary multi-objective optimisation algorithms and a local search algorithm on such landscapes by means of automated algorithm configuration. Our systematic analysis yields various insights into the behaviour and competitiveness of these algorithms and reveals that, particularly, the omni-optimizer algorithm and iterated Pareto local search yield strong, complementary, performance. These findings facilitate the case for automated algorithm selection, which we also investigate in this paper.

Optimization algorithms contain parameters that greatly influence their behavior. Finding the right settings for parameters through automated algorithm configuration has become a critical component of designing competitive algorithms. While traditional offline configurators tackle this problem by finding one configuration that works well for a set of instances, instance-specific algorithm configuration utilizes features of the instances to provide configurations that are tailored to each instance to maximize performance. We provide the first instance-specific algorithm configurator based on deep reinforcement learning that can be used in general algorithm configuration settings. Our method is able to handle large, mixed discrete and continuous search spaces and only requires a small number of instances for training. We can show that our configurator provides improvements over the state-of-the-art instance-specific configurators ISAC and Hydra on a wide range of problem domains.

We propose a versatile problem generator, called RandOptGen, for creating diverse and complex mixed-variable optimization problems, including single- and multi-objective problems. The generator implements a tree-based structure where decision variables from continuous, integer, and categorical domains are transformed into complex objectives using arbitrary mathematical operators. It ensures the feasibility of the generated problems through a validation process by, e.g., verifying that the objective spaces lie within predefined bounds and that multi-objective problems exhibit meaningful trade-offs, characterized by a well-formed Pareto front of the generated multi-objective problems.

Exploratory landscape analysis and fitness landscape analysis in general have given valuable insight into problem hardness understanding as well as facilitating algorithm design and endeavors such as automated algorithm selection and configuration. These techniques have largely been limited to search spaces of a single domain. In this work, we provide the means to compute exploratory landscape features for mixed-variable problems where the decision space is a mixture of continuous, binary, integer, and categorical variables. This is achieved by introducing a preprocessing scheme which needs to be incorporated into the process of exploratory landscape analysis feature generation. To highlight the merit of our approach for practical applications, we design and conduct an automated algorithm selection study based on a hyperparameter optimization benchmark suite and our preprocessing scheme. Our trained algorithm selector is able to close the gap between the single best and the virtual best solver by 57.5% over all benchmark problems.

In this article, we address the problem of computing all locally optimal solutions of a given multiobjective problem whose images are sufficiently close to the Pareto front. Such$\epsilon $-locally optimal solutions are particularly interesting in the context of multiobjective multimodal optimization (MMO). To accomplish this task, we first define a new set of interest,$L_{Q,\epsilon }$, that is strongly related to the recently proposed set of$\epsilon $-acceptable solutions. Next, we propose a new unbounded archiver,$ArchiveUpdateL_{Q,\epsilon }$, aiming to capture$L_{Q,\epsilon }$in the limit. This archiver can in principle be used in combination with any multiobjective evolutionary algorithm (MOEA). Further, we equip numerous MOEAs with$ArchiveUpdateL_{Q,\epsilon }$, investigate their performances across several benchmark functions, and compare the enhanced MOEAs with their archive-free counterparts. For our experiments, we utilize the well-established metrics HV, IGDX, and$\Delta _{p}$. Additionally, we propose and use a new performance indicator,$I_{\mathrm { EDR}}$, which results in comparable performances but which is applicable to problems defined in higher dimensions (in particular in decision variable space).

The herein proposed Python package pflacco provides a set of numerical features to characterize single-objective continuous and constrained optimization problems. Thereby, pflacco addresses two major challenges in the area of optimization. Firstly, it provides the means to develop an understanding of a given problem instance, which is crucial for designing, selecting, or configuring optimization algorithms in general. Secondly, these numerical features can be utilized in the research streams of automated algorithm selection and configuration. While the majority of these landscape features are already available in the R package flacco, our Python implementation offers these tools to an even wider audience and thereby promotes research interests and novel avenues in the area of optimization.

Artificial benchmark functions are commonly used in optimization research because of their ability to rapidly evaluate potential solutions, making them a preferred substitute for real-world problems. However, these benchmark functions have faced criticism for their limited resemblance to real-world problems. In response, recent research has focused on automatically generating new benchmark functions for areas where established test suites are inadequate. These approaches have limitations, such as the difficulty of generating new benchmark functions that exhibit exploratory landscape analysis (ELA) features beyond those of existing benchmarks.

Abstract Single-objective continuous optimization can be challenging, especially when dealing with multimodal problems. This work sheds light on the effects that multi-objective optimization may have in the single-objective space. For this purpose, we examine the inner mechanisms of the recently developed sophisticated local search procedure SOMOGSA. This method solves multimodal single-objective continuous optimization problems based on first expanding the problem with an additional objective (e.g., a sphere function) to the bi-objective domain and subsequently exploiting local structures of the resulting landscapes. Our study particularly focuses on the sensitivity of this multiobjectivization approach w.r.t. (1) the parametrization of the artificial second objective, as well as (2) the position of the initial starting points in the search space. As SOMOGSA is a modular framework for encapsulating local search, we integrate Nelder–Mead local search as optimizer in the respective module and compare the performance of the resulting hybrid local search to its original single-objective counterpart. We show that the SOMOGSA framework can significantly boost local search by multiobjectivization. Hence, combined with more sophisticated local search and metaheuristics, this may help solve highly multimodal optimization problems in the future.

Exploratory Landscape Analysis is a powerful technique for numerically characterizing landscapes of single-objective continuous optimization problems. Landscape insights are crucial both for problem understanding as well as for assessing benchmark set diversity and composition. Despite the irrefutable usefulness of these features, they suffer from their own ailments and downsides. Hence, in this work we provide a collection of different approaches to characterize optimization landscapes. Similar to conventional landscape features, we require a small initial sample. However, instead of computing features based on that sample, we develop alternative representations of the original sample. These range from point clouds to 2D images and, therefore, are entirely feature-free. We demonstrate and validate our devised methods on the BBOB testbed and predict, with the help of Deep Learning, the high-level, expert-based landscape properties such as the degree of multimodality and the existence of funnel structures. The quality of our approaches is on par with methods relying on the traditional landscape features. Thereby, we provide an exciting new perspective on every research area which utilizes problem information such as problem understanding and algorithm design as well as automated algorithm configuration and selection.

Abstract Hyperparameter optimization (HPO) is a key component of machine learning models for achieving peak predictive performance. While numerous methods and algorithms for HPO have been proposed over the last years, little progress has been made in illuminating and examining the actual structure of these black-box optimization problems. Exploratory landscape analysis (ELA) subsumes a set of techniques that can be used to gain knowledge about properties of unknown optimization problems. In this paper, we evaluate the performance of five different black-box optimizers on 30 HPO problems, which consist of two-, three- and five-dimensional continuous search spaces of the XGBoost learner trained on 10 different data sets. This is contrasted with the performance of the same optimizers evaluated on 360 problem instances from the black-box optimization benchmark (BBOB). We then compute ELA features on the HPO and BBOB problems and examine similarities and differences. A cluster analysis of the HPO and BBOB problems in ELA feature space allows us to identify how the HPO problems compare to the BBOB problems on a structural meta-level. We identify a subset of BBOB problems that are close to the HPO problems in ELA feature space and show that optimizer performance is comparably similar on these two sets of benchmark problems. We highlight open challenges of ELA for HPO and discuss potential directions of future research and applications.

Classic automated algorithm selection (AS) for (combinatorial) optimization problems heavily relies on so-called instance features, i.e., numerical characteristics of the problem at hand ideally extracted with computationally low-demanding routines. For the traveling salesperson problem (TSP) a plethora of features have been suggested. Most of these features are, if at all, only normalized imprecisely raising the issue of feature values being strongly affected by the instance size. Such artifacts may have detrimental effects on algorithm selection models. We propose a normalization for two feature groups which stood out in multiple AS studies on the TSP: (a) features based on a minimum spanning tree (MST) and (b) a k-nearest neighbor graph (NNG) transformation of the input instance. To this end we theoretically derive minimum and maximum values for properties of MSTs and k-NNGs of Euclidean graphs. We analyze the differences in feature space between normalized versions of these features and their unnormalized counterparts. Our empirical investigations on various TSP benchmark sets point out that the feature scaling succeeds in eliminating the effect of the instance size. Eventually, a proof-of-concept AS-study shows promising results: models trained with normalized features tend to outperform those trained with the respective vanilla features.

We consider a dynamic bi-objective vehicle routing problem, where a subset of customers ask for service over time. Therein, the distance traveled by a single vehicle and the number of unserved dynamic requests is minimized by a dynamic evolutionary multi-objective algorithm (DEMOA), which operates on discrete time windows (eras). A decision is made at each era by a decision-maker, thus any decision depends on irreversible decisions made in foregoing eras. To understand effects of sequences of decision-making and interactions/dependencies between decisions made, we conduct a series of experiments. More precisely, we fix a set of decision-maker preferences D and the number of eras ntand analyze all |D|ntcombinations of decision-maker options. We find that for random uniform instances (a) the final selected solutions mainly depend on the final decision and not on the decision history, (b) solutions are quite robust with respect to the number of unvisited dynamic customers, and (c) solutions of the dynamic approach can even dominate solutions obtained by a clairvoyant EMOA. In contrast, for instances with clustered customers, we observe a strong dependency on decision-making history as well as more variance in solution diversity.

The Traveling-Salesperson-Problem (TSP) is arguably one of the best-known NP-hard combinatorial optimization problems. The two sophisticated heuristic solvers LKH and EAX and respective (restart) variants manage to calculate close-to optimal or even optimal solutions, also for large instances with several thousand nodes in reasonable time. In this work we extend existing benchmarking studies by addressing anytime behaviour of inexact TSP solvers based on empirical runtime distributions leading to an increased understanding of solver behaviour and the respective relation to problem hardness. It turns out that performance ranking of solvers is highly dependent on the focused approximation quality. Insights on intersection points of performances offer huge potential for the construction of hybridized solvers depending on instance features. Moreover, instance features tailored to anytime performance and corresponding performance indicators will highly improve automated algorithm selection models by including comprehensive information on solver quality.

In practice, e.g. in delivery and service scenarios, Vehicle-Routing-Problems (VRPs) often imply repeated decision making on dynamic customer requests. As in classical VRPs, tours have to be planned short while the number of serviced customers has to be maximized at the same time resulting in a multi-objective problem. Beyond that, however, dynamic requests lead to the need for re-planning of not yet realized tour parts, while already realized tour parts are irreversible. In this paper we study this type of bi-objective dynamic VRP including sequential decision making and concurrent realization of decisions. We adopt a recently proposed Dynamic Evolutionary Multi-Objective Algorithm (DEMOA) for a related VRP problem and extend it to the more realistic (here considered) scenario of multiple vehicles. We empirically show that our DEMOA is competitive with a multi-vehicle offline and clairvoyant variant of the proposed DEMOA as well as with the dynamic single-vehicle approach proposed earlier.

Artificial neural networks in general and deep learning networks in particular established themselves as popular and powerful machine learning algorithms. While the often tremendous sizes of these networks are beneficial when solving complex tasks, the tremendous number of parameters also causes such networks to be vulnerable to malicious behavior such as adversarial perturbations. These perturbations can change a model's classification decision. Moreover, while single-step adversaries can easily be transferred from network to network, the transfer of more powerful multi-step adversaries has - usually - been rather difficult.In this work, we introduce a method for generating strong adversaries that can easily (and frequently) be transferred between different models. This method is then used to generate a large set of adversaries, based on which the effects of selected defense methods are experimentally assessed. At last, we introduce a novel, simple, yet effective approach to enhance the resilience of neural networks against adversaries and benchmark it against established defense methods. In contrast to the already existing methods, our proposed defense approach is much more efficient as it only requires a single additional forward-pass to achieve comparable performance results.

Evolutionary algorithms have successfully been applied to evolve problem instances that exhibit a significant difference in performance for a given algorithm or a pair of algorithms inter alia for the Traveling Salesperson Problem (TSP). Creating a large variety of instances is crucial for successful applications in the blooming field of algorithm selection. In this paper, we introduce new and creative mutation operators for evolving instances of the TSP. We show that adopting those operators in an evolutionary algorithm allows for the generation of benchmark sets with highly desirable properties: (1) novelty by clear visual distinction to established benchmark sets in the field, (2) visual and quantitative diversity in the space of TSP problem characteristics, and (3) significant performance differences with respect to the restart versions of heuristic state-of-the-art TSP solvers EAX and LKH. The important aspect of diversity is addressed and achieved solely by the proposed mutation operators and not enforced by explicit diversity preservation.

Automated algorithm selection and configuration are key enabling approaches for improving the state of the art in solving a broad range of important problems, by exploiting performance complementarity between multiple algorithms for the same problem (selection) and by realising the latent performance potential in parameterised algorithms (configuration). Compared to traditional, manual approaches, these techniques are not only more efficient and rely less on human expertise, but also provide a more principled basis for algorithm selection and configuration, enable fairer comparisons between algorithms, and facilitate new insights into which algorithmic techniques and components work best and under which circumstances.Work on automated algorithm selection, configuration, and related areas spans multiple, weakly connected communities, including artificial intelligence, evolutionary computation, mathematical optimisation and operations research. This special issue follows a Dagstuhl seminar on the same topic, held in October 2016, and is intended as a further step toward creating synergy between those communities.For this special issue, we selected, from a substantial number of submissions, seven papers that jointly cover a broad range of topics and approaches, including various methods for algorithm selection and configuration, search landscape analysis, software engineering aspects, as well as applications to prominent discrete and continuous, single- and multiobjective optimisation problems.The survey paper by Kerschke et al. provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Unlike earlier surveys, it covers applications to discrete and continuous problems; it also situates algorithm selection in the context of a wide spectrum of conceptually related approaches, such as algorithm configuration, scheduling, and portfolio selection, and discusses open challenges.Alyaha and Rowe present a study of landscape characteristics for three NP-hard combinatorial optimisation problems: number partitioning and two variants of knapsack problems. A comparative analysis of landscapes induced by different neighbourhood operators, penalty functions, and problem parameters led to improved problem understanding and to a heuristic for selecting the most appropriate local search operator.The paper by Saalem et al. introduces an approach for assessing the effectiveness of exploratory landscape features with respect to their impact on the performance of algorithm selection methods. A model-based framework for continuous black-box problem comparison using Gaussian process (GP) regression is presented, leading to a flexible surrogate model for problem landscapes. The GP substantially facilitates problem comparison while efficiently measuring model quality.Kerschke et al. present an automated algorithm selection method for single-objective continuous black-box optimisation problems, based on sophisticated exploratory landscape analysis and machine learning techniques. The efficiency of the selector is illustrated on the Black-Box Optimisation Benchmark (BBOB), by improving the performance over the single best solver from a representative set of solvers by a factor of two on average.In the paper by Wessing et al., an improved initialisation procedure for a well-known automated configuration procedure, irace, is shown to outperform classical uniform sampling of algorithm configurations. Techniques from the design and analysis of computer experiments are applied, that is, several Latin hypercube sampling methods that are able to handle categorical and numerical parameters that may be conditional (nested) on the value of other (branching) parameters.Blot et al. investigate the automatic configuration of multiobjective local search algorithms for permutation problems—specifically, for the bi-objective permutation flowshop and travelling salesman problems. They consider two performance metrics as configuration objectives and study several approaches for automated configuration according to these two objectives, presenting clear evidence that multiobjective algorithms are best configured using a multiobjective configurator.Finally, Swan et al. introduce a new approach to the automation of the design of metaheuristics, the so-called Automated Open-Closed Principle (AOCP), which addresses the problem of state dependencies between configurable components in flexible algorithm frameworks. AOCP extends current approaches to automated algorithm configuration by offering a principled software engineering approach to ensure the automated assembly of algorithms from an extensible palette of components.By now, automated algorithm selection and configuration are mature research areas, as witnessed not only by the sophistication of the methods and the success of their many applications, but also by a large and fast-growing body of literature. Still, we see much room for further work, spanning the gamut from theoretical to empirical studies, from new methodology to applications. We hope that this special issue will inspire interest and future work in these dynamic and exciting research areas.

It has long been observed that for practically any computational problem that has been intensely studied, different instances are best solved using different algorithms. This is particularly pronounced for computationally hard problems, where in most cases, no single algorithm defines the state of the art; instead, there is a set of algorithms with complementary strengths. This performance complementarity can be exploited in various ways, one of which is based on the idea of selecting, from a set of given algorithms, for each problem instance to be solved the one expected to perform best. The task of automatically selecting an algorithm from a given set is known as the per-instance algorithm selection problem and has been intensely studied over the past 15 years, leading to major improvements in the state of the art in solving a growing number of discrete combinatorial problems, including propositional satisfiability and AI planning. Per-instance algorithm selection also shows much promise for boosting performance in solving continuous and mixed discrete/continuous optimisation problems. This survey provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Different from earlier work, it covers applications to discrete and continuous problems, and discusses algorithm selection in context with conceptually related approaches, such as algorithm configuration, scheduling, or portfolio selection. Since informative and cheaply computable problem instance features provide the basis for effective per-instance algorithm selection systems, we also provide an overview of such features for discrete and continuous problems. Finally, we provide perspectives on future work in the area and discuss a number of open research challenges.

In this article, we build upon previous work on designing informative and efficient Exploratory Landscape Analysis features for characterizing problems' landscapes and show their effectiveness in automatically constructing algorithm selection models in continuous black-box optimization problems. Focusing on algorithm performance results of the COCO platform of several years, we construct a representative set of high-performing complementary solvers and present an algorithm selection model that, compared to the portfolio's single best solver, on average requires less than half of the resources for solving a given problem. Therefore, there is a huge gain in efficiency compared to classical ensemble methods combined with an increased insight into problem characteristics and algorithm properties by using informative features. The model acts on the assumption that the function set of the Black-Box Optimization Benchmark is representative enough for practical applications. The model allows for selecting the best suited optimization algorithm within the considered set for unseen problems prior to the optimization itself based on a small sample of function evaluations. Note that such a sample can even be reused for the initial population of an evolutionary (optimization) algorithm so that even the feature costs become negligible.

We continue recent work on the definition of multimodality in multiobjective optimization (MO) and the introduction of a test bed for multimodal MO problems. This goes beyond well-known diversity maintenance approaches but instead focuses on the landscape topology induced by the objective functions. More general multimodal MO problems are considered by allowing ellipsoid contours for single-objective subproblems. An experimental analysis compares two MO algorithms, one that explicitly relies on hypervolume gradient approximation, and one that is based on local search, both on a selection of generated example problems. We do not focus on performance but on the interaction induced by the problems and algorithms, which can be described by means of specific characteristics explicitly designed for the multimodal MO setting. Furthermore, we widen the scope of our analysis by additionally applying visualization techniques in the decision space. This strengthens and extends the foundations for Exploratory Landscape Analysis (ELA) in MO.

We analyze the effects of including local search techniques into a multi-objective evolutionary algorithm for solving a bi-objective orienteering problem with a single vehicle while the two conflicting objectives are minimization of travel time and maximization of the number of visited customer locations. Experiments are based on a large set of specifically designed problem instances with different characteristics and it is shown that local search techniques focusing on one of the objectives only improve the performance of the evolutionary algorithm in terms of both objectives. The analysis also shows that local search techniques are capable of sending locally optimal solutions to foremost fronts of the multi-objective optimization process, and that these solutions then become the leading factors of the evolutionary process.

The Travelling Salesperson Problem (TSP) is one of the best-studied NP-hard problems. Over the years, many different solution approaches and solvers have been developed. For the first time, we directly compare five state-of-the-art inexact solvers-namely, LKH, EAX, restart variants of those, and MAOS-on a large set of well-known benchmark instances and demonstrate complementary performance, in that different instances may be solved most effectively by different algorithms. We leverage this complementarity to build an algorithm selector, which selects the best TSP solver on a per-instance basis and thus achieves significantly improved performance compared to the single best solver, representing an advance in the state of the art in solving the Euclidean TSP. Our in-depth analysis of the selectors provides insight into what drives this performance improvement.

Exploratory Landscape Analysis (ELA) aims at understanding characteristics of single-objective continuous (black-box) optimization problems in an automated way. Moreover, the approach provides the basis for constructing algorithm selection models for unseen problem instances. Recently, it has gained increasing attention and numerical features have been designed by various research groups. This paper introduces the R-Package FLACCO which makes all relevant features available in a unified framework together with efficient helper functions. Moreover, a case study which gives perspectives to ELA for multi-objective optimization problems is presented.

When selecting the best suited algorithm for an unknown optimization problem, it is useful to possess some a priori knowledge of the problem at hand. In the context of single-objective, continuous optimization problems such knowledge can be retrieved by means of Exploratory Landscape Analysis (ELA), which automatically identifies properties of a landscape, e.g., the so-called funnel structures, based on an initial sample. In this paper, we extract the relevant features (for detecting funnels) out of a large set of landscape features when only given a small initial sample consisting of 50 x D observations, where D is the number of decision space dimensions. This is already in the range of the start population sizes of many evolutionary algorithms. The new Multiple Peaks Model Generator (MPM2) is used for training the classifier, and the approach is then very successfully validated on the Black-Box Optimization Benchmark (BBOB) and a subset of the CEC 2013 niching competition problems.

In single-objective optimization different optimization strategies exist depending on the structure and characteristics of the underlying problem. In particular, the presence of so-called funnels in multimodal problems offers the possibility of applying techniques exploiting the global structure of the function. The recently proposed Exploratory Landscape Analysis approach automatically identifies problem characteristics based on a moderately small initial sample of the objective function and proved to be effective for algorithm selection problems in continuous black-box optimization. In this paper, specific features for detecting funnel structures are introduced and combined with the existing ones in order to classify optimization problems regarding the funnel property. The effectiveness of the approach is shown by experiments on specifically generated test instances and validation experiments on standard benchmark problems.

We evaluate the performance of a multi-objective evolutionary algorithm on a class of dynamic routing problems with a single vehicle. In particular we focus on relating algorithmic performance to the most prominent characteristics of problem instances. The routing problem considers two types of customers: mandatory customers must be visited whereas optional customers do not necessarily have to be visited. Moreover, mandatory customers are known prior to the start of the tour whereas optional customers request for service at later points in time with the vehicle already being on its way. The multi-objective optimization problem then results as maximizing the number of visited customers while simultaneously minimizing total travel time. As an a-posteriori evaluation tool, the evolutionary algorithm aims at approximating the related Pareto set for specifically designed benchmarking instances differing in terms of number of customers, geographical layout, fraction of mandatory customers, and request times of optional customers. Conceptional and experimental comparisons to online heuristic procedures are provided.

In multiobjective optimization, set-based performance indicators are commonly used to assess the quality of a Pareto front approximation. Based on the scalarization obtained by these indicators, a performance comparison of multiobjective optimization algorithms becomes possible. The R2 and the hypervolume (HV) indicator represent two recommended approaches which have shown a correlated behavior in recent empirical studies. Whereas the HV indicator has been comprehensively analyzed in the last years, almost no studies on the R2 indicator exist. In this extended version of our previous conference paper, we thus perform a comprehensive investigation of the properties of the R2 indicator in a theoretical and empirical way. The influence of the number and distribution of the weight vectors on the optimal distribution of μ solutions is analyzed. Based on a comparative analysis, specific characteristics and differences of the R2 and HV indicator are presented. Furthermore, the R2 indicator is integrated into an indicator-based steady-state evolutionary multiobjective optimization algorithm (EMOA). It is shown that the so-called R2-EMOA can accurately approximate the optimal distribution of μ solutions regarding R2.

We present methods to answer two basic questions that arise when benchmarking optimization algorithms. The first one is: which algorithm is the "best" one? and the second one is: which algorithm should I use for my real-world problem? Both are connected and neither is easy to answer. We present a theoretical framework for designing and analyzing the raw data of such benchmark experiments. This represents a first step in answering the aforementioned questions. The 2009 and 2010 BBOB benchmark results are analyzed by means of this framework and we derive insight regarding the answers to the two questions. Furthermore, we discuss how to properly aggregate rankings from algorithm evaluations on individual problems into a consensus, its theoretical background and which common pitfalls should be avoided. Finally, we address the grouping of test problems into sets with similar optimizer rankings and investigate whether these are reflected by already proposed test problem characteristics, finding that this is not always the case.

Here we address the problem of computing finite size Hausdorff approximations of the Pareto front of four-objective optimization problems by means of evolutionary computing. Since many applications desire an approximation evenly spread along the Pareto front and approximations that are good in the Hausdorff sense are typically evenly spread along the Pareto front we consider three different evolutionary multi-objective algorithms tailored to that purpose, where two of them are based on the Part and Selection Algorithm (PSA). Finally, we present some numerical results indicating the strength of the novel methods.

Understanding the behaviour of well-known algorithms for classical NP-hard optimisation problems is still a difficult task. With this paper, we contribute to this research direction and carry out a feature based comparison of local search and the well-known Christofides approximation algorithm for the Traveling Salesperson Problem. We use an evolutionary algorithm approach to construct easy and hard instances for the Christofides algorithm, where we measure hardness in terms of approximation ratio. Our results point out important features and lead to hard and easy instances for this famous algorithm. Furthermore, our cross-comparison gives new insights on the complementary benefits of the different approaches.

The steady supply of new optimization methods makes the algorithm selection problem (ASP) an increasingly pressing and challenging task, specially for real-world black-box optimization problems. The introduced approach considers the ASP as a cost-sensitive classification task which is based on Exploratory Landscape Analysis. Low-level features gathered by systematic sampling of the function on the feasible set are used to predict a well-performing algorithm out of a given portfolio. Example-specific label costs are defined by the expected runtime of each candidate algorithm. We use one-sided support vector regression to solve this learning problem. The approach is illustrated by means of the optimization problems and algorithms of the BBOB'09/10 workshop.

In multiobjective optimization, set-based performance indicators are commonly used to assess the quality of a Pareto front approximation. Based on the scalarization obtained by these indicators, a performance comparison of multiobjective optimization algorithms becomes possible. The R2 and the Hypervolume (HV) indicator represent two recommended approaches which have shown a correlated behavior in recent empirical studies. Whereas the HV indicator has been comprehensively analyzed in the last years, almost no studies on the R2 indicator exist. In this paper, we thus perform a comprehensive investigation of the properties of the R2 indicator in a theoretical and empirical way. The influence of the number and distribution of the weight vectors on the optimal distribution of μ solutions is analyzed. Based on a comparative analysis, specific characteristics and differences of the R2 and HV indicator are presented.

Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.

Exploratory Landscape Analysis subsumes a number of techniques employed to obtain knowledge about the properties of an unknown optimization problem, especially insofar as these properties are important for the performance of optimization algorithms. Where in a first attempt, one could rely on high-level features designed by experts, we approach the problem from a different angle here, namely by using relatively cheap low-level computer generated features. Interestingly, very few features are needed to separate the BBOB problem groups and also for relating a problem to high-level, expert designed features, paving the way for automatic algorithm selection.

Choosing and tuning an optimization procedure for a given class of nonlinear optimization problems is not an easy task. One way to proceed is to consider this as a tournament, where each procedure will compete in different `disciplines'. Here, disciplines could either be different functions, which we want to optimize, or specific performance measures of the optimization procedure. We would then be interested in the algorithm that performs best in a majority of cases or whose average performance is maximal. We will focus on evolutionary multiobjective optimization algorithms (EMOA), and will present a novel approach to the design and analysis of evolutionary multiobjective benchmark experiments based on similar work from the context of machine learning. We focus on deriving a consensus among several benchmarks over different test problems and illustrate the methodology by reanalyzing the results of the CEC 2007 EMOA competition.

The design and application of termination criteria has become an important aspect in evolutionary multi-objective optimization. Online convergence detection (OCD) determines when further generations are no longer promising based on statistical tests on a set of performance indicators. The behavior of OCD mainly depends on two parameters, the number of preceding generations considered in the statistical tests and the desired variance limit. In this paper, guidelines for selecting appropriate combinations of these parameters are empirically derived based on design-of-experiment methods. Furthermore, a variant of OCD is introduced which directly operates on the hypervolume indicator - the internal measure of the SMS-EMOA. This allows a separated analysis of the variance criterion and reduces the complexity of OCD. Based on the experimental design, a systematic comparison with the classical OCD approach is performed and differences between the appropriate parameterizations of both variants are highlighted.

In this paper, a concept for efficiently approximating the practically relevant regions of the Pareto front (PF) is introduced. Instead of the original objectives, desirability functions (DFs) of the objectives are optimized, which express the preferences of the decision maker. The original problem formulation and the optimization algorithm do not have to be modified. DFs map an objective to the domain [0, 1] and nonlinearly increase with better objective quality. By means of this mapping, values of different objectives and units become comparable. A biased distribution of the solutions in the PF approximation based on different scalings of the objectives is prevented. Thus, we propose the integration of DFs into theS-metric selection evolutionary multiobjective algorithm. The transformation ensures the meaning of the hypervolumes internally computed. Furthermore, it is shown that the reference point for the hypervolume calculation can be set intuitively. The approach is analyzed using standard test problems. Moreover, a practical validation by means of the optimization of a turning process is performed.

Industry applications of multiobjective optimization problems mostly are characterized by the demand for high quality solutions on the one hand. On the other hand an optimization result is desired which at any rate meets the time constraints for the evolutionary multiobjective algorithms (EMOA). The handling of this trade-off is a frequently discussed issue in multiobjective evolutionary optimization.

Noisy environments are a challenging task for multiobjective evolutionary algorithms. The algorithms may be trapped in local optima or even become a random search in the decision and objective space. In the course of the paper the classical definition of Pareto-dominance is enhanced subject to noisy objective functions in order to make the evolutionary search process more robust and to generate a reliable Pareto front. At each point in the decision space the objective functions are evaluated a fixed number of times and the convex hull of the objective function vectors is computed. Expectation is associated with the median of the objective function values while uncertainty is reflected by the average distance of the median in each dimension to the points defining the convex hull. By combining these two indicators a new concept of Pareto-dominance is set up. An implementation in NSGA-II and application to test problems show a gain in robustness and search quality.

In this paper, two approaches for estimating the generation in which a multi-objective evolutionary algorithm (MOEA) shows statistically significant signs of convergence are introduced. A set-based perspective is taken where convergence is measured by performance indicators. The proposed techniques fulfill the requirements of proper statistical assessment on the one hand and efficient optimisation for real-world problems on the other hand. The first approach accounts for the stochastic nature of the MOEA by repeating the optimisation runs for increasing generation numbers and analysing the performance indicators using statistical tools. This technique results in a very robust offline procedure. Moreover, an online convergence detection method is introduced as well. This method automatically stops the MOEA when either the variance of the performance indicators falls below a specified threshold or a stagnation of their overall trend is detected. Both methods are analysed and compared for two MOEA and on different classes of benchmark functions. It is shown that the methods successfully operate on all stated problems needing less function evaluations while preserving good approximation quality at the same time.

Multi-objective evolutionary algorithms (MOEAs) are generally designed to find a well spread Pareto-front approximation. Often, only a small section of this front may be of practical interest. Desirability functions (DFs) are able to describe user preferences intuitively. Furthermore, DFs can be attached to any fitness function easily. This way, desirability functions can help in guiding MOEAs without introducing additional restrictions or changes to the algorithm. The application of noisy fitness functions is not straight forward but relevant to many real-world problems. Therefore, a variant of Harrington's one-sided desirability function using expectations is introduced which takes noise into account. A deterministic strategy as well as the XSGA-II are used in combination with DF to solve a noisy Binh problem and a noisy cost estimation problem for turning processes.