Stochastic multi-objective optimization (SMOOP) requires ranking multivariate distributions; yet, most empirical studies perform scalarization, which loses information and is unreliable. Based on the optimal transport theory, we introduce the center-outward q-dominance relation and prove it implies strong first-order stochastic dominance (FSD). Also, we develop an empirical test procedure based on q-dominance, and derive an explicit sample size threshold, n(δ), to control the Type I error. We verify the usefulness of our approach in two scenarios: (1) as a ranking method in hyperparameter tuning; (2) as a selection method in multi-objective optimization algorithms. For the former, we analyze the final stochastic Pareto sets of seven multi-objective hyperparameter tuners on the YAHPO-MO benchmark tasks with q-dominance, which allows us to compare these tuners when the expected hypervolume indicator (HVI, the most common performance metric) of the Pareto sets becomes indistinguishable. For the latter, we replace the mean value-based selection in the NSGA-II algorithm with q-dominance, which shows a superior convergence rate on noise-augmented ZDT benchmark problems. These results establish center-outward q-dominance as a principled, tractable foundation for seeking truly stochastically dominant solutions for SMOOPs.

Structural bias (SB) refers to systematic preferences of an optimisation algorithm for particular regions of the search space that arise independently of the objective function. While SB has been studied extensively in single-objective optimisation, its role in multi-objective optimisation remains largely unexplored. This is problematic, as dominance relations, diversity preservation and Pareto-based selection mechanisms may introduce or amplify structural effects.

Bayesian optimization (BO) is a class of algorithms for optimizing expensive black-box functions, but designing effective BO algorithms remains a manual, expertise-driven task. Recent advancements in Large Language Models (LLMs) have opened new avenues for automating scientific discovery, including the automatic design of optimization algorithms. While prior work has used LLMs within optimization loops or to generate non-BO algorithms, we tackle a new challenge: Using LLMs to automatically generate full BO algorithm code. Our framework uses an evolution strategy to guide an LLM in generating Python code that preserves the key components of BO algorithms: An initial design, a surrogate model, and an acquisition function. The LLM is prompted to produce multiple candidate algorithms, which are evaluated on the BBOB test suite from the COCO platform. Based on their performance, top candidates are selected, combined, and mutated via controlled prompt variations, enabling iterative refinement. Despite no additional fine-tuning, the LLM-generated algorithms outperform state-of-the-art BO baselines in 19 (out of 24) BBOB functions in dimension 5 and generalize well to higher dimensions and different tasks. This work demonstrates that LLMs can serve as algorithmic co-designers, offering a new paradigm for automating BO development and accelerating the discovery of novel algorithmic combinations.

Exploratory Landscape Analysis (ELA) provides numerical features for characterizing black-box optimization problems. In high-dimensional settings, however, ELA suffers from sparsity effects, high estimator variance, and the prohibitive cost of computing several feature classes. Dimensionality reduction has therefore been proposed as a way to make ELA applicable in such settings, but it remains unclear whether features computed in reduced spaces still reflect intrinsic properties of the original landscape.

Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's κ of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices.

We present a novel approach for constructing discrete optimization benchmarks that enables fine-grained control over problem properties, and such benchmarks can facilitate analyzing discrete algorithm behaviors. We build benchmark problems based on a set of block functions, where each block function maps a subset of variables to a real value. Problems are instantiated through a set of block functions, weight factors, and an adjacency graph representing the dependency among the block functions. Through analyzing intermediate block values, our framework allows to analyze algorithm behavior not only in the objective space but also at the level of variable representations in the obtained solutions. This capacity is particularly useful for analyzing discrete heuristics in large-scale multi-modal problems, thereby enhancing the practical relevance of benchmark studies. We demonstrate how the proposed approach can inspire the related work in self-adaptation and diversity control in evolutionary algorithms. Moreover, we explain that the proposed benchmark design enables explicit control over problem properties, supporting research in broader domains such as dynamic algorithm configuration and multi-objective optimization.

The advent of Large Language Models (LLMs) has opened new frontiers in automated algorithm design, giving rise to numerous powerful methods. However, these approaches retain critical limitations: they require extensive evaluation of the target problem to guide the search process, making them impractical for real-world optimization tasks, where each evaluation consumes substantial computational resources. This research proposes an innovative and efficient framework that decouples algorithm discovery from high-cost evaluation. Our core innovation lies in combining a Genetic Programming (GP) function generator with an LLM-driven evolutionary algorithm designer. The evolutionary direction of the GP-based function generator is guided by the similarity between the landscape characteristics of generated proxy functions and those of real-world problems, ensuring that algorithms discovered via proxy functions exhibit comparable performance on real-world problems. Our method enables deep exploration of the algorithmic space before final validation while avoiding costly real-world evaluations. We validate the framework's efficacy across multiple real-world problems, demonstrating its ability to discover high-performance algorithms while substantially reducing expensive evaluations. This approach shows a path to apply LLM-based automated algorithm design to computationally intensive real-world optimization challenges.

The practical deployment of Federated Learning (FL) on resource-constrained devices is fundamentally limited by the high cost of training large models and the instability caused by heterogeneous (non-IID) client data. Conventional pruning methods often treat data heterogeneity as a problem to be mitigated. In this work, we introduce a paradigm shift: we reframe client diversity as a feature to be harnessed. We propose AutoFLIP, a framework that begins not with training, but with a one-time federated loss exploration. During this phase, clients collaboratively build a map of the collective loss landscape, using their diverse data to reveal the problem's essential structure. This shared intelligence then guides an adaptive pruning strategy that is dynamically refined by client agreement throughout training. This approach allows AutoFLIP to identify robust and efficient sub-networks from the outset. Our extensive experiments show that AutoFLIP reduces computational overhead by an average of 52% and communication costs by over 65% while simultaneously achieving state-of-the-art accuracy in challenging non-IID settings.

NWO

Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. However, constructing a high-quality surrogate model often demands extensive data acquisition. A solution to this issue is to transfer pre-trained surrogate models for new tasks, provided that certain invariances exist between tasks. This study focuses on transferring non-differentiable surrogate models (e.g., random forests) from a source function to a target function, where we assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target. Previous research attempts to tackle this challenge for differentiable models, e.g., Gaussian process regression, which minimizes the empirical loss on the transfer data by tuning the affine transformations. In this paper, we extend the previous work to the random forest and assess its effectiveness on a widely-used artificial problem set - Black-Box Optimization Benchmark (BBOB) testbed, and on four real-world transfer learning problems. The results highlight the significant practical advantages of the proposed method, particularly in reducing both the data requirements and computational costs of training surrogate models for complex real-world scenarios.

Abstract Indicator-based (multi-objective) diversity optimization aims at finding a set of near (Pareto)optimal solutions that maximizes a diversity indicator, where diversity is typically interpreted as the number of essentially different solutions. Whereas, in the first diversity-oriented evolutionary multi-objective optimization algorithm, the NOAH algorithm by Ulrich and Thiele, the Solow Polasky Diversity (SP Diversity, also known as Magnitude [1]) served as a metric, other diversity indicators could be considered. We examine the parameter-free Max-Min Diversity and the Riesz $$s$$ s -Energy, which features uniformly distributed solution sets. Focusing on multi-objective diversity optimization, we discuss different diversity indicators from the perspective of indicator-based evolutionary algorithms with multiple objectives. We examine theoretical, computational, and practical properties of these indicators, such as monotonicity in species, twinning, monotonicity in distance, strict monotonicity in distance, uniformity of maximizing point sets, computational effort for a set of size $$n$$ n , single-point contributions, subset selection, and submodularity. We present new theorems—including a proof of the NP-hardness of the Riesz $$s$$ s -Energy Subset Selection Problem—and consolidate existing results from the literature. In the experiments, we apply these indicators in the NOAH algorithm to analyze search dynamics via an example. We study how optimizing one indicator impacts others and propose NOAH-specific modifications for the Max-Min indicator.

The mutation process in evolution strategies has been interlinked with the normal distribution since its inception. Many lines of reasoning have been given for this strong dependency, ranging from maximum entropy arguments to the need for isotropy. However, some theoretical results suggest that other distributions might lead to similar local convergence properties. This paper empirically shows that a wide range of evolutionary strategies, from the (1+1)-ES to CMA-ES, show comparable optimization performance when using a mutation distribution other than the standard Gaussian. Replacing it with, e.g., uniformly distributed mutations, does not deteriorate the performance of ES, when using the default adaptation mechanism for the strategy parameters. We observe that these results hold not only for the sphere model but also for a wider range of benchmark problems.

Variational quantum algorithms, such as the Recursive Quantum Approximate Optimization Algorithm (RQAOA), have become increasingly popular, offering promising avenues for employing Noisy Intermediate-Scale Quantum devices to address challenging combinatorial optimization tasks like the maximum cut problem. In this study, we utilize an evolutionary algorithm equipped with a unique fitness function. This approach targets hard maximum cut instances within the latent space of a Graph Autoencoder, identifying those that pose significant challenges or are particularly tractable for RQAOA, in contrast to the classic Goemans and Williamson algorithm. Our findings not only delineate the distinct capabilities and limitations of each algorithm but also expand our understanding of RQAOA's operational limits. Furthermore, the diverse set of graphs we have generated serves as a crucial benchmarking asset, emphasizing the need for more advanced algorithms to tackle combinatorial optimization challenges. Additionally, our results pave the way for new avenues in graph generation research, offering exciting opportunities for future explorations.

Large Language Models (LLMs) have demonstrated great promise in generating code, especially when used inside an evolutionary computation framework to iteratively optimize the generated algorithms. However, in some cases they fail to generate competitive algorithms or the code optimization stalls, and we are left with no recourse because of a lack of understanding of the generation process and generated codes. We present a novel approach to mitigate this problem by enabling users to analyze the generated codes inside the evolutionary process and how they evolve over repeated prompting of the LLM. We show results for three benchmark problem classes and demonstrate novel insights. In particular, LLMs tend to generate more complex code with repeated prompting, but additional complexity can hurt algorithmic performance in some cases. Different LLMs have different coding "styles" and generated code tends to be dissimilar to other LLMs. These two findings suggest that using different LLMs inside the code evolution frameworks might produce higher performing code than using only one LLM.

Combining large language models with evolutionary computation algorithms represents a promising research direction leveraging the remarkable generative and in-context learning capabilities of LLMs with the strengths of evolutionary algorithms. In this work, we present EvoCAD, a method for generating computer-aided design (CAD) objects through their symbolic representations using vision language models and evolutionary optimization. Our method samples multiple CAD objects, which are then optimized using an evolutionary approach with vision language and reasoning language models. We assess our method using GPT-4V and GPT-4o, evaluating it on the CAD-Prompt benchmark dataset and comparing it to prior methods. Additionally, we introduce two new metrics based on topological properties defined by the Euler characteristic, which capture a form of semantic similarity between 3D objects. Our results demonstrate that EvoCAD outperforms previous approaches on multiple metrics, particularly in generating topologically correct objects, which can be efficiently evaluated using our two novel metrics that complement existing spatial metrics.

Combining natural language and geometric shapes is an emerging research area with multiple applications in robotics and language-assisted design. A crucial task in this domain is object referent identification, which involves selecting a 3D object given a textual description of the target. Variability in language descriptions and spatial relationships of 3D objects makes this a complex task, increasing the need to better understand the behavior of neural network models in this domain. However, limited research has been conducted in this area. Specifically, when a model makes an incorrect prediction despite being provided with a seemingly correct object description, practitioners are left wondering: "Why is the model wrong?". In this work, we present a method answering this question by generating counterfactual examples. Our method takes a misclassified sample, which includes two objects and a text description, and generates an alternative yet similar formulation that would have resulted in a correct prediction by the model. We have evaluated our approach with data from the ShapeTalk dataset along with three distinct models. Our counterfactual examples maintain the structure of the original description, are semantically similar and meaningful. They reveal weaknesses in the description, model bias and enhance the understanding of the models behavior. Theses insights help practitioners to better interact with systems as well as engineers to improve models.

In recent years, simulation-based optimization methods for analog integrated circuit design parameters optimization (a.k.a sizing) have attracted extensive research interest. Currently, researchers primarily focus on developing efficient algorithms while paying little attention to decision spaces. This work focuses on the decision space, aiming to improve the efficiency and usability of the parameters optimization task. linear and dynamic circuits. We also handle circuit constraints by directly fine-tuning search bounds. Second, taking the high-fidelity EKV model, we demonstrate the unique characteristics of the electrical design space and prove that a bijective relationship exists between the two decision spaces. Third, we propose a multi-form (MF) optimization framework that simultaneously optimizes the physical design space and electrical design space. This framework avoids the choice of decision space and enhances the algorithm’s optimization efficiency by transferring candidate solutions between two decision spaces. Also, we propose to solve the MF optimization task with Bayesian optimization and population-based algorithms. The proposed sizing framework is verified on three typical analog circuit sizing tasks: single-objective, multi-objective, and yield optimization problems. The result and ablation study show that the proposed framework consistently achieves better results compared to traditional single-space optimization methods, with significantly fewer iterations.

Large language models (LLMs), such as GPT-4 have demonstrated their ability to understand natural language and generate complex code snippets. This article introduces a novel LLM evolutionary algorithm (LLaMEA) framework, leveraging GPT models for the automated generation and refinement of algorithms. Given a set of criteria and a task definition (the search space), LLaMEA iteratively generates, mutates, and selects algorithms based on performance metrics and feedback from runtime evaluations. This framework offers a unique approach to generating optimized algorithms without requiring extensive prior expertise. We show how this framework can be used to generate novel closed box metaheuristic optimization algorithms for box-constrained, continuous optimization problems automatically. LLaMEA generates multiple algorithms that outperform state-of-the-art optimization algorithms (covariance matrix adaptation evolution strategy and differential evolution) on the 5-D closed box optimization benchmark (BBOB). The algorithms also show competitive performance on the 10- and 20-D instances of the test functions, although they have not seen such instances during the automated generation process. The results demonstrate the feasibility of the framework and identify future directions for automated generation and optimization of algorithms via LLMs.

Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This article presents a novel approach that we call explainable benchmarking. We introduce the IOHxplainer software library, for systematic analysing the performance of various optimization algorithms and the impact of their different components and hyperparameters. We showcase the methodology in the context of two modular optimization implementations. Through this library, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, aiming to improve future benchmarking and algorithm design practices.

Choosing a set of benchmark problems is often a key component of any empirical evaluation of iterative optimization heuristics. In continuous, single-objective optimization, several sets of problems have become widespread, including the well-established BBOB suite. While this suite is designed to enable rigorous benchmarking, it is also commonly used for testing methods such as algorithm selection, which the suite was never designed around. We present the MA-BBOB function generator, which uses the BBOB suite as component functions in an affine combination. In this work, we describe the full procedure to create these affine combinations and highlight the tradeoffs of several design decisions, specifically the choice to place the optimum uniformly at random in the domain. We then illustrate how this generator can be used to gain more low-level insight into the function landscapes through the use of exploratory landscape analysis. Finally, we show a potential use-case of MA-BBOB in generating a wide set of training and testing data for algorithm selectors. Using this setup, we show that the basic scheme of using a set of landscape features to predict the best algorithm does not lead to optimal results, and that an algorithm selector trained purely on the BBOB functions generalizes poorly to the affine combinations.

Solving multi-objective vehicle routing problem with time windows (VRPTW) is a challenge in last-mile logistics. While traditional methods have predominantly focused on minimizing travel distances and vehicle numbers, recent innovations acknowledge the need for a more refined balance across various objectives. These advancements incorporate a range of heuristic algorithms, blending hybrid techniques that merge genetic algorithms with local search strategies, and employing advanced methodologies such as particle swarm optimization and multi-objective evolutionary algorithms based on decomposition (MOEA/D). Such developments signify a shift towards more effective solutions for VRPTW. In response to these challenges, this paper introduces a novel approach that integrates MOEA/D with K-means clustering for generating efficient solutions. In our approach, we refine the tri-objective VRPTW efficiency by establishing a criterion that prioritizes customer sequences based on their spatial proximity and angular alignment relative to cluster centers and depot, utilizing an integrated metric of distance and cosine angles, which can enhance the local search process and effectively balance the three objectives in VRPTW. Experimental results on Solomon's 56 VRPTW 100-customer dataset instances demonstrate the superiority of our approach.

Chance-constrained problems involve stochastic components in the constraints which can be violated with a small probability. We investigate the impact of different types of chance constraints on the performance of iterative search algorithms and study the classical maximum coverage problem in graphs with chance constraints. Our goal is to evolve reliable chance constraint settings for a given graph where the performance of algorithms differs significantly not just in expectation but with high confidence. This allows to better learn and understand how different types of algorithms can deal with different types of constraint settings and supports automatic algorithm selection. We develop an evolutionary algorithm that provides sets of chance constraints that differentiate the performance of two stochastic search algorithms with high confidence. We initially use traditional approximation ratio as the fitness function of (1+1) EA to evolve instances, which shows inadequacy to generate reliable instances. To address this issue, we introduce a new measure to calculate the performance difference for two algorithms, which considers variances of performance ratios. Our experiments show that our approach is highly successful in solving the instability issue of the performance ratios and leads to evolving reliable sets of chance constraints with significantly different performance for various types of algorithms.

Symbolic regression (SR) poses a significant challenge for randomized search heuristics due to its reliance on the synthesis of expressions for input-output mappings. Although traditional genetic programming (GP) algorithms have achieved success in various domains, they exhibit limited performance when tree-based representations are used for SR. To address these limitations, we introduce a novel SR approach called Fourier Tree Growing (FTG) that draws insights from functional analysis. This new perspective enables us to perform optimization directly in a different space, thus avoiding intricate symbolic expressions. Our proposed algorithm exhibits significant performance improvements over traditional GP methods on a range of classical one-dimensional benchmarking problems. To identify and explain the limiting factors of GP and FTG, we perform experiments on a large-scale polynomials benchmark with high-order polynomials up to degree 100. To the best of the authors' knowledge, this work represents the pioneering application of functional analysis in addressing SR problems. The superior performance of the proposed algorithm and insights into the limitations of GP open the way for further advancing GP for SR and related areas of explainable machine learning.

The reference implementation of Cartesian Genetic Programming (CGP) was written in the C programming language. C inherently follows a procedural programming paradigm, which entails challenges in providing a reusable and scalable implementation model for complex structures and methods. Moreover, due to the limiting factors of C, the reference implementation of CGP does not provide a generic framework and is therefore restricted to a set of predefined evaluation types. Besides the reference implementation, we also observe that other existing implementations are limited with respect to the features provided. In this work, we therefore propose the first version of a modern C++ implementation of CGP that pursues object-oriented design and generic programming paradigm to provide an efficient implementation model that can facilitate the discovery of new problem domains and the implementation of complex advanced methods that have been proposed for CGP over time. With the proposal of our new implementation, we aim to generally promote interpretability, accessibility and reproducibility in the field of CGP.

Surrogate models are widely applied in many scenarios to replace expensive executions of real-world procedures. Training a high-quality surrogate model often requires many sample points, which can be costly to obtain. We would amortize this cost if we could reuse already-trained surrogates in future tasks, provided certain invariances are retained across tasks. This paper studies transferring a surrogate model trained on a source function to a target function using a small data set. As a first step, we consider the following invariance: the domains of the source and target functions are related by an unknown affine transformation. We propose to parameterize the surrogate of the source with an affine transformation and optimize it w.r.t. an empirical loss measured with a small transfer data set sampled on the target. We select all functions from the well-known black-box optimization benchmark (BBOB) as the source and artificially generate the target with affine transformation sampled u.a.r. We experiment with a commonly used surrogate model, Gaussian process regression, where results show that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch with the transfer data set.

The number of proposed iterative optimization heuristics is growing steadily, and with this growth, there have been many points of discussion within the wider community. One particular criticism that is raised towards many new algorithms is their focus on metaphors used to present the method, rather than emphasizing their potential algorithmic contributions. Several studies into popular metaphor-based algorithms have highlighted these problems, even showcasing algorithms that are functionally equivalent to older existing methods. Unfortunately, this detailed approach is not scalable to the whole set of metaphor-based algorithms. Because of this, we investigate ways in which benchmarking can shed light on these algorithms. To this end, we run a set of 294 algorithm implementations on the BBOB function suite. We investigate how the choice of the budget, the performance measure, or other aspects of experimental design impact the comparison of these algorithms. Our results emphasize why benchmarking is a key step in expanding our understanding of the algorithm space, and what challenges still need to be overcome to fully gauge the potential improvements to the state-of-the-art hiding behind the metaphors.

Parameter control has succeeded in accelerating the convergence process of evolutionary algorithms. While empirical and theoretical studies have shed light on the behavior of algorithms for single-objective optimization, little is known about how self-adaptation influences multi-objective evolutionary algorithms. In this work, we contribute (1) extensive experimental analysis of the Global Simple Evolutionary Multi-objective Algorithm (GSEMO) variants on classic problems, such as OneMinMax, LOTZ, COCZ, and (2) a novel version of GSEMO with self-adjusting mutation rates.

The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is one of the most successful examples of a derandomized evolution strategy. However, it still relies on randomly sampling offspring, which can be done via a uniform distribution and subsequently transforming into the required Gaussian. Previous work has shown that replacing this uniform sampling with a low-discrepancy sampler, such as Halton or Sobol sequences, can improve performance over a wide set of problems. We show that iterating through small, fixed sets of low-discrepancy points can still perform better than the default uniform distribution. Moreover, using only 128 points throughout the search is sufficient to closely approximate the empirical performance of using the complete pseudorandom sequence up to dimensionality 40 on the BBOB benchmark. For lower dimensionalities (below 10), we find that using as little as 32 unique low discrepancy points performs similar or better than uniform sampling. In 2D, for which w e have highly optimized low discrepancy samples available, we demonstrate that using these points yields the highest empirical performance and requires only 16 samples to improve over uniform sampling. Overall, we establish a clear relation between the L2 discrepancy of the used point set and the empirical performance of the CMA-ES.

When benchmarking optimization heuristics, we need to take care to avoid an algorithm exploiting biases in the construction of the used problems. One way in which this might be done is by providing different versions of each problem but with transformations applied to ensure the algorithms are equipped with mechanisms for successfully tackling a range of problems. In this paper, we investigate several of these problem transformations and show how they influence the low-level landscape features of problems from the Congress on Evolutionary Computation 2022 benchmark suite. Our results highlight that even relatively small transformations can significantly alter the measured landscape features. This poses a wider question of what properties we want to preserve when creating problem transformations, and how to measure them fairly.

Abstract In landscape-aware algorithm selection problem, the effectiveness of feature-based predictive models strongly depends on the representativeness of training data for practical applications. In this work, we investigate the potential of randomly generated functions (RGF) for the model training, which cover a much more diverse set of optimization problem classes compared to the widely-used black-box optimization benchmarking (BBOB) suite. Correspondingly, we focus on automated algorithm configuration (AAC), that is, selecting the best suited algorithm and fine-tuning its hyperparameters based on the landscape features of problem instances. Precisely, we analyze the performance of dense neural network (NN) models in handling the multi-output mixed regression and classification tasks using different training data sets, such as RGF and many-affine BBOB (MA-BBOB) functions. Based on our results on the BBOB functions in 5d and 20d, near optimal configurations can be identified using the proposed approach, which can most of the time outperform the off-the-shelf default configuration considered by practitioners with limited knowledge about AAC. Furthermore, the predicted configurations are competitive against the single best solver in many cases. Overall, configurations with better performance can be best identified by using NN models trained on a combination of RGF and MA-BBOB functions.

Time-series anomaly detection plays an important role in engineering processes, like development, manufacturing and other operations involving dynamic systems. These processes can greatly benefit from advances in the field, as state-of-the-art approaches may aid in cases involving, for example, highly dimensional data. To provide the reader with understanding of the terminology, this survey introduces a novel taxonomy where a distinction between online and offline, and training and inference is made. Additionally, it presents the most popular data sets and evaluation metrics used in the literature, as well as a detailed analysis. Furthermore, this survey provides an extensive overview of the state-of-the-art model-based online semi- and unsupervised anomaly detection approaches for multivariate time-series data, categorising them into different model families and other properties. The biggest research challenge revolves around benchmarking, as currently there is no reliable way to compare different approaches against one another. This problem is two-fold: on the one hand, public data sets suffers from at least one fundamental flaw, while on the other hand, there is a lack of intuitive and representative evaluation metrics in the field. Moreover, the way most publications choose a detection threshold disregards real-world conditions, which hinders the application in the real world. To allow for tangible advances in the field, these issues must be addressed in future work.

We present IOHexperimenter, the experimentation module of the IOHprofiler project. IOHexperimenter aims at providing an easy-to-use and customizable toolbox for benchmarking iterative optimization heuristics such as local search, evolutionary and genetic algorithms, and Bayesian optimization techniques. IOHexperimenter can be used as a stand-alone tool or as part of a benchmarking pipeline that uses other modules of the IOHprofiler environment. IOHexperimenter provides an efficient interface between optimization problems and their solvers while allowing for granular logging of the optimization process. Its logs are fully compatible with existing tools for interactive data analysis, which significantly speeds up the deployment of a benchmarking pipeline. The main components of IOHexperimenter are the environment to build customized problem suites and the various logging options that allow users to steer the granularity of the data records.

Abstract As restricted quantum computers become available, research focuses on finding meaningful applications. For example, in quantum machine learning, a special type of quantum circuit called a quantum neural network is one of the most investigated approaches. However, we know little about suitable circuit architectures or important model hyperparameters for a given task. In this work, we apply the functional ANOVA framework to the quantum neural network architectures to analyze which of the quantum machine learning hyperparameters are most influential for their predictive performance. We restrict our study to 7 open-source datasets from the OpenML-CC18 classification benchmark, which are small enough for simulations on quantum hardware with fewer than 20 qubits. Using this framework, three main levels of importance were identified, confirming expected patterns and revealing new insights. For instance, the learning rate is identified as the most important hyperparameter on all datasets, whereas the particular choice of entangling gates used is found to be the least important on all except for one dataset. In addition to identifying the relevant hyperparameters, for each of them, we also learned data-driven priors based on values that perform well on previously seen datasets, which can then be used to steer hyperparameter optimization processes. We utilize these priors in the hyperparameter optimization method hyperband and show that these improve performance against uniform sampling across all datasets by, on average, $$0.53 \%$$ 0.53 % , up to $$6.11 \%$$ 6.11 % , in cross-validation accuracy. We also demonstrate that such improvements hold on average regardless of the configuration hyperband is run with. Our work introduces new methodologies for studying quantum machine learning models toward quantum model selection in practice. All research code is made publicly available.

Solving real-world engineering optimization problems, such as automotive crashworthiness optimization, is extremely challenging, because the problem characteristics are oftentimes not well understood. Furthermore, typical hyperparameter optimization (HPO) approaches that require a large function evaluation budget are computationally hindered, if the function evaluation is expensive, for example, requires finite element (FE) simulation runs. In this article, we propose an approach to characterize real-world expensive black-box optimization problems using the exploratory landscape analysis (ELA). Based on these landscape characteristics, we can identify test functions that are fast-to-evaluate and representative for HPO purposes. Focusing on 20 problem instances from automotive crashworthiness optimization, our results reveal that these 20 crashworthiness problems exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used black-box optimization benchmarking (BBOB) problem set. In fact, these 20 problem instances belong to problem classes that are distinct from the BBOB test functions based on the clustering results. Further analysis indicates that, as far as the ELA features concern, they are most similar to problem classes of tree-based test functions. By analyzing the performance of two optimization algorithms with different hyperparameters, namely the covariance matrix adaptation evolutionary strategy (CMA-ES) and Bayesian optimization (BO), we show that the tree-based test functions are indeed representative in terms of predicting the algorithm performances. Following this, such scalable and fast-to-evaluate tree-based test functions have promising potential for automated design of an optimization algorithm for specific real-world problem classes.

Submodular functions play a key role in the area of optimization as they allow to model many real-world problems that face diminishing returns. Evolutionary algorithms have been shown to obtain strong theoretical performance guarantees for a wide class of submodular problems under various types of constraints while clearly outperforming standard greedy approximation algorithms. This paper introduces a setup for benchmarking algorithms for submodular optimization problems with the aim to provide researchers with a framework to enhance and compare the performance of new algorithms for submodular problems. The focus is on the development of iterative search algorithms such as evolutionary algorithms with the implementation provided and integrated into IOHprofiler which allows for tracking and comparing the progress and performance of iterative search algorithms. We present a range of submodular optimization problems that have been integrated into IOHprofiler and show how the setup can be used for analyzing and comparing iterative search algorithms in various settings.

Abstract A building spatial design (BSD) determines external and internal walls and ceilings of a building. The design space has a hierarchical structure, in which decisions on the existence or non-existence of spatial components determine the existence of variables related to these spaces, such as sizing and angles. In the optimization of BSDs it is envisioned to optimize various performance indicators from multiple disciplines in concert, such as structural, functional, thermal, and daylight performance. Existing representations of design spaces suffer from severe limitations, such as only representing orthogonal designs or representing the structures in parametric superstructure, allowing only for limited design variations. This paper proposes prism nets - a new way of representing the search space of BSDs based on triangulations defining space filling collections of triangular prisms that can be combined via coloring parameters to spaces. Prism nets can accommodate for non-orthogonal designs and are flexible in terms of topological variations. We follow the guidelines for representation and operator design proposed in the framework of metric-based evolutionary algorithms. The main contribution of the paper is a detailed discussion of the search space representation and corresponding mutation operators. Moreover, a proof of concept example demonstrates the integration into multi-objective evolutionary algorithms and provides first results on a simple, but reproducible, benchmark problem.

The objective value of an ill-conditioned function may significantly change with a minor shift of the argument in the search space. Ill-conditioned functions do not have at all or exhibit very few hints towards better solutions and, thus, they are usually difficult to optimize with randomized search heuristics. However, problems that emerge in practical applications are likely to be formulated as ill-conditioned functions, as often Euclidean metric is used to measure distance in the search space. At the same time, it may be possible to use domain-specific knowledge to define such a metric in the search space so that the function stops being ill-conditioned. We consider finite search spaces and propose two mutation operators that leverage such metric to optimize the function more efficiently. The first operator assumes prior knowledge about the distance, the second operator uses the distance as a black box. Those operators apply an estimation of distribution algorithm to find the best mutant according to the defined function, which employs the given metric. For pseudo-Boolean and integer optimization problems, we experimentally show that both mutation operators speed up the search on most of the functions when applied in considered evolutionary algorithms and random local search. Moreover, those operators can be applied in any randomized search heuristic which uses perturbations. However, our mutation operators increase wall-clock time and so are helpful in practice when distance is (much) cheaper to compute than the real objective function.

Just over a decade ago, the first comprehensive review on the state of benchmarking in Genetic Programming (GP) analyzed the mismatch between the problems that are used to test the performance of GP systems and real-world problems. Since then, several benchmark suites in major GP problem domains have been proposed over time, filling some of the major gaps. In the framework of the first review about the state of benchmarking in GP, logic synthesis (LS) was classified as one of the major GP problem domains. However, a diverse and accessible benchmark suite for LS is still missing. In this work, we propose a benchmark suite for LS that covers different types of Boolean functions that are commonly used in the field of GP. We analyze the complexity of the proposed benchmark by using popular complexity measures that are commonly used to classify and characterize Boolean functions and digital circuits.

The domain of an optimization problem is seen as one of its most important characteristics. In particular, the distinction between continuous and discrete optimization is rather impactful. Based on this, the optimizing algorithm, analyzing method, and more are specified. However, in practice, no problem is ever truly continuous. Whether this is caused by computing limits or more tangible properties of the problem, most variables have a finite resolution.

New contributions in the field of iterative optimisation heuristics are often made in an iterative manner. Novel algorithmic ideas are not proposed in isolation, but usually as extensions of a preexisting algorithm. Although these contributions are often compared to the base algorithm, it is challenging to make fair comparisons between larger sets of algorithm variants. This happens because even small changes in the experimental setup, parameter settings, or implementation details can cause results to become incomparable. Modular algorithms offer a way to overcome these challenges. By implementing the algorithmic modifications into a common framework, many algorithm variants can be compared, while ensuring that implementation details match in all versions.

A clear need for automatic anomaly detection applied to automotive testing has emerged as more and more attention is paid to the data recorded and manual evaluation by humans reaches its capacity. Such real-world data is massive, diverse, multivariate and temporal in nature, therefore requiring modelling of the testee behaviour. We propose a variational autoencoder with multi-head attention (MA-VAE), which, when trained on unlabelled data, not only provides very few false positives but also manages to detect the majority of the anomalies presented. In addition to that, the approach offers a novel way to avoid the bypass phenomenon, an undesirable behaviour investigated in literature. Lastly, the approach also introduces a new method to remap individual windows to a continuous time series. The results are presented in the context of a real-world in-dustrial data set and several experiments are undertaken to further investigate certain aspects of the proposed model. When configured pro perly, it is 9% of the time wrong when an anomaly is flagged and discovers 67% of the anomalies present. Also, MA-VAE has the potential to perform well with only a fraction of the training and validation subset, however, to extract it, a more sophisticated threshold estimation method is required.

The algorithm selection problem is of paramount importance in achieving high-quality results while minimizing computational effort, especially when dealing with expensive black-box optimization problems. In this paper, we address this challenge by using randomly generated artificial functions that mimic the landscape characteristics of the original problem while being inexpensive to evaluate. The similarity between the artificial function and the original problem is quantified using Exploratory Landscape Analysis. We demonstrate a significant performance improvement on five real-world vehicle dynamics problems by transferring the parameters of the Covariance Matrix Adaptation Evolution Strategy tuned to these artificial functions. We provide a complete set of simulated values of braking distance for fully enumerated 2D design spaces of all five real-world optimization problems. So, replication of our results and benchmarking directly on the real-world problems is possible. Beyond the scope of this paper, this data can be used as a benchmarking set for multi-objective optimization with up to five objectives.

The performance of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is significantly affected by the selection of the specific CMA-ES variant and the parameter values used. Furthermore, optimal CMA-ES parameter configurations vary across different problem landscapes, making the task of tuning CMA-ES to a specific optimization problem a challenging mixed-integer optimization problem. In recent years, several advanced algorithms have been developed to address this problem, including the Sequential Model-based Algorithm Configuration (SMAC) and the Tree-structured Parzen Estimator (TPE). In this study, we propose a novel approach for tuning CMA-ES by leveraging CMA-ES itself. Therefore, we combine the modular CMA-ES implementation with the margin extension to handle mixed-integer optimization problems. We show that CMA-ES can not only compete with SMAC and TPE but also outperform them in terms of wall clock time.

Thirty years, 1993-2023, is a huge time frame in science. We address some major developments in the field of evolutionary algorithms, with applications in parameter optimization, over these 30 years. These include the covariance matrix adaptation evolution strategy and some fast-growing fields such as multimodal optimization, surrogate-assisted optimization, multiobjective optimization, and automated algorithm design. Moreover, we also discuss particle swarm optimization and differential evolution, which did not exist 30 years ago, either. One of the key arguments made in the paper is that we need fewer algorithms, not more, which, however, is the current trend through continuously claiming paradigms from nature that are suggested to be useful as new optimization algorithms. Moreover, we argue that we need proper benchmarking procedures to sort out whether a newly proposed algorithm is useful or not. We also briefly discuss automated algorithm design approaches, including configurable algorithm design frameworks, as the proposed next step toward designing optimization algorithms automatically, rather than by hand.

Abstract Optimization problems with multiple objectives and many input variables inherit challenges from both large-scale optimization and multi-objective optimization. To solve the problems, decomposition and transformation methods are frequently used. In this study, an improved control variable analysis is proposed based on dominance and diversity in Pareto optimization. Further, the decomposition method is used in a cooperative coevolution framework with orthogonal sampling mutation. The algorithm’s performances are compared against the weighted optimization framework. The results show that the proposed decomposition method has much better accuracy compared to the traditional method. The results also show that the cooperative coevolution framework with a good grouping is very competitive. Additionally, the number of search directions in orthogonal sampling can be easily configured. A small number of search directions will reduce the search space greatly while also restricting the area that can be explored and vice versa.

Oftentimes the characteristics of real-world engineering optimization problems are not well understood. In this paper, we introduce an approach for characterizing highly nonlinear and Finite Element (FE) simulation-based engineering optimization problems, focusing on ten representative problem instances from automotive crashworthiness optimization. By computing characteristic Exploratory Landscape Analysis (ELA) features, we show that these ten crashworthiness problem instances exhibit landscape features different from classical optimization benchmark test suites, such as the widely-used Black-Box Optimization Benchmarking (BBOB) problem set. Using clustering approaches, we demonstrate that these ten problem instances are clearly distinct from the BBOB test functions. Further analysis of the crashworthiness problem instances reveal that, as far as ELA concerns, they are most similar to a class of artificially generated functions. We identify such artificially generated functions and propose to use them as scalable and fast-to-evaluate representatives of the real-world problems. Such artificially generated functions could be used for the automated design of an optimization algorithm for specific real-world problem classes.

The stochastic nature of iterative optimization heuristics leads to inherently noisy performance measurements. Since these measurements are often gathered once and then used repeatedly, the number of collected samples will have a significant impact on the reliability of algorithm comparisons. We show that care should be taken when making decisions based on limited data. Particularly, we show that the number of runs used in many benchmarking studies, e.g., the default value of 15 suggested by the COCO environment, can be insufficient to reliably rank algorithms on well-known numerical optimization benchmarks.

Bayesian optimization is often used to optimize expensive black box optimization problems with long simulation times. Typically Bayesian optimization algorithms propose one solution per iteration. The downside of this strategy is the sub-optimal use of available computing power. To efficiently use the available computing power (or a number of licenses etc.) we introduce a multi-point acquisition function for parallel efficient multi-objective optimization algorithms. The multi-point acquisition function is based on the hypervolume contribution of multiple solutions simultaneously, leading to well spread solutions along the Pareto frontier. By combining this acquisition function with a constraint handling technique, multiple feasible solutions can be proposed and evaluated in parallel every iteration. The hypervolume and feasibility of the solutions can easily be estimated by using multiple cheap radial basis functions as surrogates with different configurations. The acquisition function can be used with different population sizes and even for one shot optimization. The strength and generalizability of the new acquisition function is demonstrated by optimizing a set of black box constraint multi-objective problem instances. The experiments show a huge time saving factor by using our novel multi-point acquisition function, while only marginally worsening the hypervolume after the same number of function evaluations.

Abstract We introduce the problem ofrobust subgroup discovery, i.e., finding a set of interpretable descriptions of subsets that 1) stand out with respect to one or more target attributes, 2) are statistically robust, and 3) non-redundant. Many attempts have been made to mine eitherlocallyrobust subgroups or to tackle the pattern explosion, but we are the first to address both challenges at the same time from aglobalmodelling perspective. First, we formulate the broad model class of subgroup lists, i.e., ordered sets of subgroups, for univariate and multivariate targets that can consist of nominal or numeric variables, including traditional top-1 subgroup discovery in its definition. This novel model class allows us to formalise the problem of optimal robust subgroup discovery using the Minimum Description Length (MDL) principle, where we resort to optimal Normalised Maximum Likelihood and Bayesian encodings for nominal and numeric targets, respectively. Second, finding optimal subgroup lists is NP-hard. Therefore, we propose SSD++, a greedy heuristic that finds good subgroup lists and guarantees that the most significant subgroup found according to the MDL criterion is added in each iteration. In fact, the greedy gain is shown to be equivalent to a Bayesian one-sample proportion, multinomial, or t-test between the subgroup and dataset marginal target distributions plus a multiple hypothesis testing penalty. Furthermore, we empirically show on 54 datasets that SSD++ outperforms previous subgroup discovery methods in terms of quality, generalisation on unseen data, and subgroup list size.

Benchmarking provides an essential ground base for adequately assessing and comparing evolutionary computation methods and other optimization algorithms. It allows us to gain insights into strengths and weaknesses of different existing techniques, and consequently design more efficient optimization approaches. The need for good benchmarking practices opens up a broad range of complementary research questions, arising as a byproduct of challenges encountered when optimization methods are assessed. From the selection of representative benchmark problem instances, different algorithms, and suitable performance metrics, over efficient experimentation, to a sound evaluation of the benchmark data, these research questions lie at the core of establishing a well-designed and standardized benchmarking procedure.

The choice of design representations, as of search operators, is central to the performance of evolutionary optimization algorithms, in particular, for multitask problems. The multitask approach pushes further the parallelization aspect of these algorithms by solving simultaneously multiple optimization tasks using a single population. During the search, the operators implicitly transfer knowledge between solutions to the offspring, taking advantage of potential synergies between problems to drive the solutions to optimality. Nevertheless, in order to operate on the individuals, the design space of each task has to be mapped to a common search space, which is challenging in engineering cases without clear semantic overlap between parameters. Here, we apply a 3-D point cloud autoencoder to map the representations from the Cartesian to a unified design representation: the latent space of the autoencoder. The transfer of latent space features between design representations allows the reconstruction of shapes with interpolated characteristics and maintenance of common parts, which potentially improves the performance of the designs in one or more tasks during the optimization. Compared to traditional representations for shape optimization, such as free-form deformation, the latent representation enables more representative design modifications, while keeping the baseline characteristics of the learned classes of objects. We demonstrate the efficiency of our approach in an optimization scenario where we minimize the aerodynamic drag of two different car shapes with common underbodies for cost-efficient vehicle platform design.

Finding the best configuration of algorithms’ hyperparameters for a given optimization problem is an important task in evolutionary computation. We compare in this work the results of four different hyperparameter optimization (HPO) approaches for a family of genetic algorithms (GAs) on 25 diverse pseudo-Boolean optimization (PBO) problems. More precisely, we compare previously obtained results from a grid search with those obtained from three automated configuration techniques: 1) iterated racing; 2) mixed-integer parallel-efficient global optimization (MIP-EGO); and 3) mixed-integer evolutionary strategies. Using two different cost metrics: 1) expected running time (ERT) and 2) the area under the empirical cumulative distribution function (ECDF) curve, we find that in several cases the best configurations with respect to ERT are obtained when using the area under the ECDF curve as the cost metric during the configuration process. Our results suggest that even when interested in ERT performance, it might be preferable to use anytime performance measures for the configuration task. We also observe that tuning for ERT is much more sensitive with respect to the budget that is allocated to the target algorithms.

Benchmarking and performance analysis play an important role in understanding the behaviour of iterative optimization heuristics (IOHs) such as local search algorithms, genetic and evolutionary algorithms, Bayesian optimization algorithms, etc. This task, however, involves manual setup, execution, and analysis of the experiment on an individual basis, which is laborious and can be mitigated by a generic and well-designed platform. For this purpose, we propose IOHanalyzer, a new user-friendly tool for the analysis, comparison, and visualization of performance data of IOHs. Implemented in R and C++ , IOHanalyzer is fully open source. It is available on CRAN and GitHub. IOHanalyzer provides detailed statistics about fixed-target running times and about fixed-budget performance of the benchmarked algorithms with a real-valued codomain, single-objective optimization tasks. Performance aggregation over several benchmark problems is possible, for example in the form of empirical cumulative distribution functions. Key advantages of IOHanalyzer over other performance analysis packages are its highly interactive design, which allows users to specify the performance measures, ranges, and granularity that are most useful for their experiments, and the possibility to analyze not only performance traces, but also the evolution of dynamic state parameters. IOHanalyzer can directly process performance data from the main benchmarking platforms, including the COCO platform, Nevergrad, the SOS platform, and IOHexperimenter. An R programming interface is provided for users preferring to have a finer control over the implemented functionalities.

Methods for learning on 3D point clouds became ubiquitous due to the popularization of 3D scanning technology and advances of machine learning techniques. Among these methods, point-based deep neural networks have been utilized to explore 3D designs in optimization tasks. However, engineering computer simulations require high-quality meshed models, which are challenging to automatically generate from unordered point clouds. In this work, we propose Point2FFD: A novel deep neural network for learning compact geometric representations and generating simulation-ready meshed models. Built upon an autoencoder architecture, Point2FFD learns to compress 3D point clouds into a latent design space, from which the network generates 3D polygonal meshes by selecting and deforming simulation-ready mesh templates. Through benchmark experiments, we show that our proposed network achieves comparable shape-generative performance than existing state-of-the-art point-based generative models. In real world-inspired vehicle aerodynamic optimizations, we demonstrate that Point2FFD generates simulation-ready meshes of realistic car shapes and leads to better optimized designs than the benchmarked networks.

Methods for learning and compressing high-dimensional data allow designers to generate novel and low-dimensional design representations for shape optimization problems. By using compact design spaces, global optimization algorithms require less function evaluations to characterize the problem landscape. Furthermore, data-driven representations are often domain-agnostic and independent of the user expertise, and thus potentially capture more relevant design features than a human designer would suggest. However, more factors than the dimensionality play a role in the efficiency of design representations. In this paper, we perform a comparative analysis of design representations for 3D shape optimization problems obtained with principal component analysis, kernel-principal component analysis and a 3D point cloud autoencoder, which we apply on a benchmark data set of computer aided engineering car models. We evaluate the shape-generative capabilities of these methods and show that we can modify the geometries more locally with the autoencoder than with the remaining methods. In a vehicle aerodynamic optimization framework, we verify that this property of the autoencoder representation improves the optimization performance by enabling potentially complementary degrees of freedom for the optimizer. With our study, we provide insights on the qualitative properties and quantifiable measures on the efficiency of deep neural networks as shape generative models for engineering optimization problems, as well as analyses of geometric representations for engineering optimization with evolutionary algorithms.

The imbalanced classification problem is very relevant in both academic and industrial applications. The task of finding the best machine learning model to use for a specific imbalanced dataset is complicated due to a large number of existing algorithms, each with its own hyperparameters. The Combined Algorithm Selection and Hyperparameter optimization (CASH) has been introduced to tackle both aspects at the same time. However, CASH has not been studied in detail in the class imbalance domain, where the best combination of resampling technique and classification algorithm is searched for, together with their optimized hyperparameters. Thus, we target the CASH problem for imbalanced classification. We experiment with a search space of 5 classification algorithms, 21 resampling approaches and 64 relevant hyperparameters in total. Moreover, we investigate performance of 2 well-known optimization approaches: Random search and Tree Parzen Estimators approach which is a kind of Bayesian optimization. For comparison, we also perform grid search on all combinations of resampling techniques and classification algorithms with their default hyperparameters. Our experimental results show that a Bayesian optimization approach outperforms the other approaches for CASH in this application domain.

We consider multi-solution optimization and generative models for the generation of diverse artifacts and the discovery of novel solutions. In cases where the domain's factors of variation are unknown or too complex to encode manually, generative models can provide a learned latent space to approximate these factors. When used as a search space, however, the range and diversity of possible outputs are limited to the expressivity and generative capabilities of the learned model. We compare the output diversity of a quality diversity evolutionary search performed in two different search spaces: 1) a predefined parameterized space and 2) the latent space of a variational autoencoder model. We find that the search on an explicit parametric encoding creates more diverse artifact sets than searching the latent space. A learned model is better at interpolating between known data points than at extrapolating or expanding towards unseen examples. We recommend using a generative model's latent space primarily to measure similarity between artifacts rather than for search and generation. Whenever a parametric encoding is obtainable, it should be preferred over a learned representation as it produces a higher diversity of solutions.

In this study, we analyze behaviours of the well-known CMA-ES by extracting the time-series features on its dynamic strategy parameters. An extensive experiment was conducted on twelve CMA-ES variants and 24 test problems taken from the BBOB (Black-Box Optimization Bench-marking) testbed, where we used two different cutoff times to stop those variants. We utilized the tsfresh package for extracting the features and performed the feature selection procedure using the Boruta algorithm, resulting in 32 features to distinguish either CMA-ES variants or the problems. After measuring the number of predefined targets reached by those variants, we contrive to predict those measured values on each test problem using the feature. From our analysis, we saw that the features can classify the CMA-ES variants, or the function groups decently, and show a potential for predicting the performance of those variants. We conducted a hierarchical clustering analysis on the test problems and noticed a drastic change in the clustering outcome when comparing the longer cutoff time to the shorter one, indicating a huge change in search behaviour of the algorithm. In general, we found that with longer time series, the predictive power of the time series features increase.

Differential evolution is one of the most prestigious population-based stochastic optimization algorithm for black-box problems. The performance of a differential evolution algorithm depends highly on its mutation and crossover strategy and associated control parameters. However, the determination process for the most suitable parameter setting is troublesome and time consuming. Adaptive control parameter methods that can adapt to problem landscape and optimization environment are more preferable than fixed parameter settings. This article proposes a novel adaptive parameter control approach based on learning from the optimization experiences over a set of problems. In the approach, the parameter control is modeled as a finite-horizon Markov decision process. A reinforcement learning algorithm, named policy gradient, is applied to learn an agent (i.e., parameter controller) that can provide the control parameters of a proposed differential evolution adaptively during the search procedure. The differential evolution algorithm based on the learned agent is compared against nine well-known evolutionary algorithms on the CEC'13 and CEC'17 test suites. Experimental results show that the proposed algorithm performs competitively against these compared algorithms on the test suites.

Needle electromyography (EMG) is a common technique used in clinical neurophysiology to record the electrical activity of muscles at different levels of activation. It can be used to diagnose various neurological/muscular disorders, as the EMG signals of patients with both nerve diseases (neuropathies) and muscle diseases (myopathies) differ from the signal in healthy controls. A major drawback of this examination is that it relies on visual inspection and as such, it is highly subjective and prone to errors. Based on EMG time series of 65 individuals (40 with ALS/IBM and 25 healthy), we aim to develop an automated machine-learning pipeline for the classification of EMG recordings of muscles in either disease or healthy (muscle-level). The automated pipeline consists of feature extraction, feature selection, modelling algorithm, and optimization, in which the most significant features are automatically selected from the feature space and the hyperparameters of the model are optimized by a Bayesian technique as part of the automated approach. Aside from the muscle-level approach, we also explore a patient-level approach, which uses the output of the muscle-level automated pipeline in a post-processing manner to classify patients in being either disease or healthy, based on their muscle recordings. The resulting two approaches yield an AUC score of 81.7% (muscle-level) and 81.5% (patient-level), indicating that such approaches can assist clinicians in diagnosing if a patient has a neuropathy/myopathy or is healthy.

This paper investigates whether optimisation methods with the population made up of one solution can suffer from structural bias just like their multisolution variants. Following recent results highlighting the importance of choice of strategy for handling solutions generated outside the domain, a selection of single solution methods are considered in conjunction with several such strategies. Obtained results are tested for the presence of structural bias by means of a traditional approach from literature and a newly proposed here statistical approach. These two tests are demonstrated to be not fully consistent. All tested methods are found to be structurally biased with at least one of the tested strategies. Confirming results for multisolution methods, it is such strategy that is shown to control the emergence of structural bias in single solution methods. Some of the tested methods exhibit a kind of structural bias that has not been observed before.

One of the most challenging problems in evolutionary computation is to select from its family of diverse solvers one that performs well on a given problem. This algorithm selection problem is complicated by the fact that different phases of the optimization process require different search behavior. While this can partly be controlled by the algorithm itself, there exist large differences between algorithm performance. It can therefore be beneficial to swap the configuration or even the entire algorithm during the run. Long deemed impractical, recent advances in Machine Learning and in exploratory landscape analysis give hope that this dynamic algorithm configuration (dynAC) can eventually be solved by automatically trained configuration schedules. With this work we aim at promoting research on dynAC, by introducing a simpler variant that focuses only on switching between different algorithms, not configurations. Using the rich data from the Black Box Optimization Benchmark (BBOB) platform, we show that even single-switch dynamic Algorithm selection (dynAS) can potentially result in significant performance gains. We also discuss key challenges in dynAS, and argue that the BBOB-framework can become a useful tool in overcoming these.

When faced with a specific optimization problem, deciding which algorithm to apply is always a difficult task. Not only is there a vast variety of algorithms to select from, but these algorithms are often controlled by many hyperparameters, which need to be suitably tuned in order to achieve peak performance. Usually, the problem of selecting and configuring the optimization algorithm is addressed sequentially, by first selecting a suitable algorithm and then tuning it for the application at hand. Integrated approaches, commonly known as Combined Algorithm Selection and Hyperparameter (CASH) solvers, have shown promise in several applications.

Convolutional neural network training can suffer from diverse issues like exploding or vanishing gradients, scaling-based weight space symmetry and covariant-shift. In order to address these issues, researchers develop weight regularization methods and activation normalization methods. In this work we propose a weight soft-regularization method based on the Oblique manifold. The proposed method uses a loss function which pushes each weight vector to have a norm close to one, i.e. the weight matrix is smoothly steered toward the so-called Oblique manifold. We evaluate our method on the very popular CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art architectures, namely the ResNet and wide-ResNet. Our method introduces negligible computational overhead and the results show that it is competitive to the state-of-the-art and in some cases superior to it. Additionally, the results are less sensitive to hyperparameter settings such as batch size and regularization factor.

Computational Fluid Dynamics (CFD) simulations are a very important tool for many industrial applications, such as aerodynamic optimization of engineering designs like cars shapes, airplanes parts etc. The output of such simulations, in particular the calculated flow fields, are usually very complex and hard to interpret for realistic three-dimensional real-world applications, especially if time-dependent simulations are investigated. Automated data analysis methods are warranted but a non-trivial obstacle is given by the very large dimensionality of the data. A flow field typically consists of six measurement values for each point of the computational grid in 3D space and time (velocity vector values, turbulent kinetic energy, pressure and viscosity). In this paper we address the task of extracting meaningful results in an automated manner from such high dimensional data sets. We propose deep learning methods which are capable of processing such data and which can be trained to solve relevant tasks on simulation data, i.e. predicting drag and lift forces applied on an airfoil. We also propose an adaptation of the classical hand crafted features known from computer vision to address the same problem and compare a large variety of descriptors and detectors. Finally, we compile a large dataset of 2D simulations of the flow field around airfoils which contains 16000 flow fields with which we tested and compared approaches. Our results show that the deep learning-based methods, as well as hand crafted feature based approaches, are well-capable to accurately describe the content of the CFD simulation output on the proposed dataset.

Some real-world domains, such as Agriculture and Healthcare, comprise early-stage disease indications whose recording constitutes a rare event, and yet, whose precise detection at that stage is critical. In this type of highly imbalanced classification problems, which encompass complex features, deep learning (DL) is much needed because of its strong detection capabilities. At the same time, DL is observed in practice to favor majority over minority classes and consequently suffer from inaccurate detection of the targeted early-stage indications. To simulate such scenarios, we artificially generate skewness (99% vs. 1%) for certain plant types out of the PlantVillage dataset as a basis for classification of scarce visual cues through transfer learning. By randomly and unevenly picking healthy and unhealthy samples from certain plant types to form a training set, we consider a base experiment as fine-tuning ResNet34 and VGG19 architectures and then testing the model performance on a balanced dataset of healthy and unhealthy images. We empirically observe that the initial F1 test score jumps from 0.29 to 0.95 for the minority class upon adding a final Batch Normalization (BN) layer just before the output layer in VGG19. We demonstrate that utilizing an additional BN layer before the output layer in modern CNN architectures has a considerable impact in terms of minimizing the training time and testing error for minority classes in highly imbalanced data sets. Moreover, when the final BN is employed, minimizing the loss function may not be the best way to assure a high F1 test score for minority classes in such problems. That is, the network might perform better even if it is not `confident' enough while making a prediction; leading to another discussion about why softmax output is not a good uncertainty measure for DL models. We also report on the corroboration of these findings on the ISIC Skin Cancer as well as the Wall Crack datasets.

In order to reduce the dimensionality of 3D point cloud representations, autoencoder architectures generate increasingly abstract, compressed features of the input data. Visualizing these features is central to understanding the learning process, however, while successful visualization techniques exist for neural networks applied to computer vision tasks, similar methods for geometric, especially non-Euclidean, input data are currently lacking. Hence, we propose a first-of-kind method to project the features learned by point cloud autoencoders into a 3D-space augmented with color maps. Our proposal explores the properties of 1D-convolutions, used in state-of-the art point cloud autoencoder architectures to handle the input data, which leads to an intuitive interpretation of the visualized features. Furthermore, we tackle the search for relevant co-activations in the feature space by clustering the input data in the latent space, where we explore the correspondence between network features and geometric characteristics of typical shapes of the clusters. We tested our approach with experiments on a benchmark data set, and with three different configurations of a point cloud autoencoder, where we show that the features learned by the autoencoder correlate with the occupancy of the input space by the training data.

Clinical time series are known for irregular, highly-sporadic and strongly-complex structures and are consequently difficult to model by traditional state-space models. In this paper, we investigate the potential of applying variational recurrent neural networks (VRNNs) for forecasting clinical time series extracted from electronic health records (EHRs) of patients. Variational recurrent neural networks (VRNNs) combine recurrent neural networks (RNNs) and variational inference (VI) and are state-of-the-art methods to model highly-variable sequential data such as text, speech, time series and multimedia signals in a generative fashion. We propose to incorporate multiple correlated time series to improve the forecasting of VRNNs. The selection of these correlated time series is based on the similarity of the supplementary medical information e.g., disease diagnostics, ethnicity and age etc. between the patients. We evaluate the effectiveness of utilizing such supplementary information with root mean square error (RMSE), on clinical benchmark data-set "Medical Information Mart for Intensive Care (MIMIC III)" for multi-step-ahead prediction. We further perform subjective analysis to highlight the effects of the similarity of the supplementary medical information on individual temporal features e.g., Systolic Blood Pressure (SBP), Heart Rate (HR) etc. of the patients from the same data-set. Our results clearly show that incorporating the correlated time series based on the supplementary medical information can help improving the accuracy of the VRNNs for clinical time series forecasting.

Although the anomaly detection problem can be considered as an extreme case of class imbalance problem, very few studies consider improving class imbalance classification with anomaly detection ideas. Most data-level approaches in the imbalanced learning domain aim to introduce more information to the original dataset by generating synthetic samples. However, in this paper, we gain additional information in another way, by introducing additional attributes. We propose to introduce the outlier score and four types of samples (safe, borderline, rare, outlier) as additional attributes in order to gain more information on the data characteristics and improve the classification performance. According to our experimental results, introducing additional attributes can improve the imbalanced classification performance in most cases (6 out of 7 datasets). Further study shows that this performance improvement is mainly contributed by a more accurate classification in the overlapping region of the two classes (majority and minority classes). The proposed idea of introducing additional attributes is simple to implement and can be combined with resampling techniques and other algorithmic-level approaches in the imbalanced learning domain.

Abstract KrigingorGaussian Process Regressionis applied in many fields as a non-linear regression model as well as a surrogate model in the field of evolutionary computation. However, the computational and space complexity of Kriging, that is cubic and quadratic in the number of data points respectively, becomes a major bottleneck with more and more data available nowadays. In this paper, we propose a general methodology for the complexity reduction, called cluster Kriging, where the whole data set is partitioned into smaller clusters and multiple Kriging models are built on top of them. In addition, four Kriging approximation algorithms are proposed as candidate algorithms within the new framework. Each of these algorithms can be applied to much larger data sets while maintaining the advantages and power of Kriging. The proposed algorithms are explained in detail and compared empirically against a broad set of existing state-of-the-art Kriging approximation methods on a well-defined testing framework. According to the empirical study, the proposed algorithms consistently outperform the existing algorithms. Moreover, some practical suggestions are provided for using the proposed algorithms.

Abstract Open source software projects often use issue repositories, where project contributors submit bug reports. Using these repositories, more bugs in software projects may be identified and fixed. However, the content and therefore quality of bug reports vary. In this study, we aim to understand the significance of different elements in bug reports. We interviewed 35 developers to gain insights into their perceptions on the importance of various contents in bug reports. To assess our findings, we surveyed 305 developers. The results show developers find it highly important that bug reports include crash description, reproducing steps or test cases, and stack traces. Software version, fix suggestions, code snippets, and attached contents have lower importance for software debugging. Furthermore, to evaluate the quality of currently available bug reports, we mined issue repositories of 250 most popular projects on Github. Statistical analysis on the mined issues shows that crash reproducing steps, stack traces, fix suggestions, and user contents, have statistically significant impact on bug resolution times, for ∼70%, ∼76%, ∼55%, and ∼33% of the projects. However, on avarage, over 70% of bug reports lack these elements.

The treatment of Parkinson’s Disease (PD) with Deep Brain Stimulation (DBS) can provide a constant level of motor functioning. Several patients, however, may suffer from postoperative cognitive deterioration. The DBS screening therefore includes an assessment of cognitive functioning prior to DBS surgery. However, these assessments may be influenced by factors such as fatigue or motivation and there is a need for novel biomarkers of cognitive dysfunction to complement the DBS screening. Electroencephalography (EEG) has been previously suggested to identify potential cognitive impairment in PD patients and may have utility during the DBS screening. A limited set of biomarkers (features) from the EEG has been identified for this purpose. Finding new biomarkers is time-consuming and there is no driving hypothesis on which new biomarkers may be important. Based on EEG time series of 40 DBS candidates, this research focuses on automated machine learning techniques to develop EEG-based algorithms for the evaluation of the cognitive function of PD patients. The automated pipeline consists of feature extraction, feature selection, modelling algorithm and optimization. With this approach we extract 794 features from each of the 21 EEG channels which results in a massive feature space. From this feature space the most significant features are selected and used for modelling. The hyperparameters of the model are optimized by a Bayesian technique as part of the automated approach. Aside from the automatically computed features, we also explore the use of features commonly used during clinical evaluation of the EEG, with the result that the model based on automatically computed features achieves a significant higher accuracy (84.0%). The newly identified features are potentially new biomarkers. We used the knowledge gathered from our automated approach to build a hand-crafted model resulting in an accuracy of 91.0%.

State-of-the-art Grammatical Evolution systems such as PonyGE2 have a number of hyper-parameters that control the behavior of the internal evolutionary algorithm for evolving the representations of programs. In this paper, a variant of the efficient global optimization (EGO) algorithm is applied for optimizing these hyper-parameters of the PonyGE2-system. This approach is tested on four test problems used in the Grammatical Evolution community: StringMatch, symbolic regression (the `Vladislavleva-4' problem), bank note classification and the so-called Pymax task. The experimental results show that the average performance of the GE system is improved significantly (between 25% and 168%) on all of the test problems. In addition, the resulting overall best hyper-parameter settings are substantially different from the defaults used in PonyGE2.

In this paper, a new real-world application problem, i.e., the vehicle fleet maintenance scheduling optimization problem, is defined and a specialized multi-objective evolutionary algorithm framework (grouping strategy, three vector chromosome and corresponding genetic operators) is proposed to solve the problem. State-of-the-art multi-objective evolutionary algorithms such as NSGA-III, SMS-EMOA, DI-MOEA are employed in the proposed algorithm framework to solve the problem, and their behavior is investigated. Although DI-MOEA is used the first time for a real-world application problem, its performance is better than other algorithms for some instances.

A key property underlying the success of evolutionary algorithms (EAs) is their global search behavior, which allows the algorithms to "jump" from a current state to other parts of the search space, thereby avoiding to get stuck in local optima. This property is obtained through a random choice of the radius at which offspring are sampled from previously evaluated solutions. It is well known that, thanks to this global search behavior, the probability that an EA using standard bit mutation finds a global optimum of an arbitrary function f : {0, 1}n→ ℝ tends to one as the number of function evaluations grows. This advantage over heuristics using a fixed search radius, however, comes at the cost of using non-optimal step sizes also in those regimes in which the optimal rate is stable for a long time. This downside results in significant performance losses for many standard benchmark problems. We introduce in this work a simple way to interpolate between the random global search of EAs and their deterministic counterparts which sample from a fixed radius only. To this end, we introduce normalized standard bit mutation, in which the binomial choice of the search radius is replaced by a normal distribution. Normalized standard bit mutation allows a straightforward way to control its variance, and hence the degree of randomness involved. We experiment with a self-adjusting choice of this variance, and demonstrate its effectiveness for the two classic benchmark problems LeadingOnes and OneMax. Our work thereby also touches a largely ignored question in discrete evolutionary computation: multi-dimensional parameter control.

The initial phase in real world engineering optimization and design is a process of discovery in which not all requirements can be made in advance, or are hard to formalize. Quality diversity algorithms, which produce a variety of high performing solutions, provide a unique chance to support engineers and designers in the search for what is possible and high performing. In this work we begin to answer the question how a user can interact with quality diversity and turn it into an interactive innovation aid. By modeling a user's selection it can be determined whether the optimization is drifting away from the user's preferences. The optimization is then constrained by adding a penalty to the objective function. We present an interactive quality diversity algorithm that can take into account the user's selection. The approach is evaluated in a new multimodal optimization benchmark that allows various optimization tasks to be performed. The user selection drift of the approach is compared to a state of the art alternative on both a planning and a neuroevolution control task, thereby showing its limits and possibilities.

Statistics-based Design of Experiments (DoE) methodologies are considered the gold standard in existing laboratory equipment for screening predefined experimental assays. Thus far, very little is formally known about their effectiveness in light of global optimization, particularly when compared to Evolutionary Algorithms (EAs). The current study, which was ignited by evolution-in-the-loop of functional protein expression, aims to conduct such a comparison with a focus on Combinatorial Optimization considering a dozen of decision variables, a search-space cardinality of several million combinations, under a budget of a couple of thousands of function evaluations. Due to the limited budget of evaluations, we argue that surrogate-assisted search methods become relevant for this application domain. To this end, we study a specific so-called Categorical Evolution Strategy (CatES). We systematically compare its performance, with and without being assisted by state-of-the-art surrogates, to DoE-based initialization of the search (exploiting the budget partially or entirely). Our empirical findings on noise-free benchmarks show that the surrogate-based approach is superior, as it significantly outperforms the DoE techniques and the CatES alone on the majority of the problems subject to the budget constraint. We conclude by projecting the strengths and weaknesses of EAs versus DoE, when run either directly or surrogate-aided.

In the field of evolutionary computation, one of the most challenging topics is algorithm selection. Knowing which heuristics to use for which optimization problem is key to obtaining high-quality solutions. We aim to extend this research topic by taking a first step towards a selection method for adaptive CMA-ES algorithms. We build upon the theoretical work done by van Rijn et al. [PPSN'18], in which the potential of switching between different CMA-ES variants was quantified in the context of a modular CMA-ES framework.

The technique of parallelization is a trend in the field of Bayesian global optimization (BGO) and is important for real-world applications because it can make full use of CPUs and speed up the execution times. This paper proposes a multi-point mechanism of the expected hypervolume improvement (EHVI) for multi-objective BGO (MOBGO) by the utilization of the truncated EHVI (TEHVI). The basic idea is to divide the objective space into several sub-objective spaces and then search for the optimal solutions in each sub-objective space by using the TEHVI as the infill criterion. We studied the performance of the proposed algorithm and performed comparisons with Kriging believer technique (KB) on five scientific benchmarks and a real-world application problem (i.e., a low-fidelity multi-objective airfoil optimization design). The stochastic experimental results show that the proposed algorithm performs better than the KB with respect to the hypervolume indicator, indicating that the proposed method provides an efficient parallelization technique for MOBGO.

Designing the architecture for an artificial neural network is a cumbersome task because of the numerous parameters to configure, including activation functions, layer types, and hyper-parameters. With the large number of parameters for most networks nowadays, it is intractable to find a good configuration for a given task by hand. In this paper the Mixed Integer Parallel Efficient Global Optimization (MIP-EGO) algorithm is proposed to automatically configure convolutional neural network architectures. It is shown that on several image classification tasks this approach is able to find competitive network architectures in terms of prediction accuracy, compared to the best hand-crafted ones in literature, when using only a fraction of the number of training epochs. Moreover, instead of the standard sequential evaluation in EGO, several candidate architectures are proposed and evaluated in parallel, which reduces the execution overhead significantly and leads to an efficient automation for deep neural network design.

Generating more evenly distributed samples in high dimensional search spaces is the major purpose of the recently proposed mirrored sampling technique for evolution strategies. The diversity of the mutation samples is enlarged and the convergence rate is therefore improved by the mirrored sampling. Motivated by the mirrored sampling technique, this article introduces a new derandomized sampling technique called mirrored orthogonal sampling. The performance of this new technique is both theoretically analyzed and empirically studied on the sphere function. In particular, the mirrored orthogonal sampling technique is applied to the well-known Covariance Matrix Adaptation Evolution Strategy (CMA-ES). The resulting algorithm is experimentally tested on the well-known Black-Box Optimization Benchmark (BBOB). By comparing the results from the benchmark, mirrored orthogonal sampling is found to outperform both the standard CMA-ES and its variant using mirrored sampling.

In the field of multi-objective optimization algorithms, multi-objective Bayesian Global Optimization (MOBGO) is an important branch, in addition to evolutionary multi-objective optimization algorithms. MOBGO utilizes Gaussian Process models learned from previous objective function evaluations to decide the next evaluation site by maximizing or minimizing an infill criterion. A commonly used criterion in MOBGO is the Expected Hypervolume Improvement (EHVI), which shows a good performance on a wide range of problems, with respect to exploration and exploitation. However, so far, it has been a challenge to calculate exact EHVI values efficiently. This paper proposes an efficient algorithm for the exact calculation of the EHVI for in a generic case. This efficient algorithm is based on partitioning the integration volume into a set of axis-parallel slices. Theoretically, the upper bound time complexities can be improved from previously $$O (n^2)$$ and $$O(n^3)$$ , for two- and three-objective problems respectively, to $$\varTheta (n\log n)$$ , which is asymptotically optimal. This article generalizes the scheme in higher dimensional cases by utilizing a new hyperbox decomposition technique, which is proposed by Dächert et al. (Eur J Oper Res 260(3):841–855, 2017). It also utilizes a generalization of the multilayered integration scheme that scales linearly in the number of hyperboxes of the decomposition. The speed comparison shows that the proposed algorithm in this paper significantly reduces computation time. Finally, this decomposition technique is applied in the calculation of the Probability of Improvement (PoI).

Bayesian Global Optimization algorithm is designed to optimize expensive objective functions with small evaluation budget. This algorithm employs a surrogate model and assesses the potential improvement of unseen solutions through the so-called infill-criterion. A novel infill-criterion proposed in our previous work is derived from the moment-generating function of the improvement. In contrast to other techniques, it features a continuous parameter that can be used to adjust the exploration-exploitation tradeoff smoothly. In this work, two cooling strategies (linear and exponential) are adopted to enhance the explorative behavior in the early stage of the search and the exploitative effect in the final converging stage. Moreover, the initial temperature and cooling speed are investigated on some selected multi-modal functions, showing that the good setting of those two parameters depends on the problems specifics. The proposed Bayesian optimization with cooling strategy is tested on well-known BBOB benchmark. The results shows that without tuning the initial temperature and cooling speed, the proposed approach improves the performance on a range of multi-modal functions as compared to the commonly used expected improvement criterion.

Commercial quantum processing units (QPUs) such as those made by D-Wave Systems are being increasingly used for solving complex combinatorial optimization problems. In this paper, we review a canonical NP-hard problem, the Maximum Independent Set (MIS) problem. We show how to map MIS problems to quadratic unconstrained binary optimization (QUBO) problems, and use a D-Wave 2000Q QPU to solve them. We compare the results from the D-Wave system to classical algorithms such as simulated thermal annealing and the graphical networks package NetworkX. To our knowledge, these are the first results of experiments involving arbitrarily-structured MIS inputs using a D-Wave QPU. We find that the QPU can be used as a heuristic optimizer for randomly generated inputs, but due to physical control errors, can be outperformed by simulated thermal annealing.

While much work exists on scheduling, literature in the subfield of online scheduling remains sparse. As with many problems, online scheduling has parallels with natural phenomena. Specifically, online scheduling can be seen in the division of labour among colony insects, such as ants. Although multiple different biological models exist for division of labour, the only one to have been applied in online scheduling is the reinforced threshold model, for instance in the form of the ant task allocation (ATA) algorithm. However, it is neither known how it compares to other models, nor in which applications any of them excel. This paper studies the foraging for work (FFW) model as a possible alternative. To do so, an algorithmic description of the FFW model is introduced, and it is compared with the ATA algorithm on the truck painting problem. For this problem, tasks of various types are scheduled in a flowshop with multiple identical machines in an online fashion. FFW is shown to be very effective at minimising setup time, which is incurred when switching to tasks of different types. Furthermore, this allows FFW to outperform the threshold based approaches when the scheduling environment is placed under heavy load.

Theoretical and empirical research on evolutionary computation methods complement each other by providing two fundamentally different approaches towards a better understanding of black-box optimization heuristics. In discrete optimization, both streams developed rather independently of each other, but we observe today an increasing interest in reconciling these two sub-branches. In continuous optimization, the COCO (Comparing Continuous Optimisers) benchmarking suite has established itself as an important platform that theoreticians and practitioners use to exchange research ideas and questions. No widely accepted equivalent exists in the research domain of discrete black-box optimization.

Multi-objective optimization in aerodynamic design plays an important role in capturing the trade-off and useful knowledge that would be useful for real-world design processes. In the preliminary design phase, aerodynamic designers usually have an interest in focusing the optimization process in a certain direction of interest. To this end, we propose the use of user preference multi-objective Bayesian global optimization (MOBGO) for aerodynamic design using truncated expected hypervolume improvement (TEHVI). Taking into account the apriori knowledge of objective functions, TEHVI acts as an infill criterion to search for the optimal solutions based on the Kriging models in MOBGO. In TEHVI-MOBGO, the first step is to obtain a coarse approximation of the Pareto front in order to capture the general trend and trade off using standard EHVI; following this step, TEHVI is then applied to focus the search on a defined region of interest. We demonstrate the capabilities and usefulness of TEHVI method on the design optimization of an inviscid transonic wing and a viscous transonic airfoil in order to minimize the drag coefficient and absolute value of pitching moment, which leads to a reduced fuel burn and easier control characteristic.

Using time series in-car data, this research focuses on predicting the point of impact of a low speed crash by developing an automatized machine learning approach for time series applications. After an initial data exploration, we discuss the extraction of features from time series and different ways to select the most relevant features. From 3,176 extracted features 9 are selected and used for a classification with a decision tree. To optimize the hyper-parameters of the decision tree algorithm, a randomized search with 50,000 iterations is conducted. The modeling results are graphically presented and discussed. With a final prediction accuracy of 89% (cross-validated 76%), the optimized decision tree offers great potential for utilization in vehicle insurance processing for automatized settlement of low-speed crash damages.

Computational Fluid Dynamics (CFD) simulations are able to produce complex and large outputs that accurately describe the physical properties of fluids and gases in various domains, such as air flow around a car, or the multi-phase flow inside an internal combustion engine. The simulation results, i.e. the flow fields, are often too complex to be analyzed directly. With the increasing number of simulations as well as their complexity, there is a need of automated processes that can analyze these complex outputs. In this paper, inspired by the success of convolutional neural networks (CNNs) in Computer Vision, we apply for the first time CNNs on CFD output. We show their capabilities in capturing and processing flow patterns. Furthermore, we design a novel CNN architecture tailored to the data produced by CFD simulations, as well as two conventional architectures and compare them. We propose and construct a new dataset of turbulent flow, within the application domain of steady flow around passenger cars. We use that dataset to evaluate and compare the proposed methods, on different tasks that depend on flow patterns. Finally, we compare our methods with a baseline k-nearest neighbor approach, tuned to be comparable to the state-of-the-art.

In this paper solution approaches for solving the building spatial design optimisation problem for structural and energy performance are advanced on multiple fronts. A new initialisation operator is introduced to generate an unbiased initial population for a tailored version of SMS-EMOA with problem specific operators. Improvements to the mutation operator are proposed to eliminate bias and allow mutations consisting of multiple steps. Moreover, landscape analysis is applied in order to explore the landscape of both objectives and investigate the behaviour of the mutation operator. Parameter tuning is applied with the irace package and the Mixed Integer Evolution Strategy to find improved parameter settings and explore tuning with a relatively small number of expensive evaluations. Finally, the performances of the standard and tailored SMS-EMOA algorithms with tuned parameters are compared.

Efficient Global Optimization (EGO) is an effective method to optimize expensive black-box functions and utilizes Kriging models (or Gaussian process regression) trained on a relatively small design data set. In real-world applications, such as experimental optimization, where a large data set is available, the EGO algorithm becomes computationally infeasible due to the time and space complexity of Kriging. Recently, the so-called Cluster Kriging methods have been proposed to reduce such complexities for the big data, where data sets are clustered and Kriging models are built on each cluster. Furthermore, Kriging models are combined in an optimal way for the prediction. In addition, we analyze the Cluster Kriging landscape to adopt the existing infill-criteria, e.g., the expected improvement. The approach is tested on selected global optimization problems. It is shown by the empirical studies that this approach significantly reduces the CPU time of the EGO algorithm while maintaining the convergence rate of the algorithm.

In the past years, quite a number of algorithmic extensions of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) have been proposed. These extensions define a large algorithm design space, but relatively little is known about the performance of most of these variations and the interaction between them.

Bayesian Optimization or Efficient Global Optimization (EGO) is a global search strategy that is designed for expensive black-box functions. In this algorithm, a statistical model (usually the Gaussian process model) is constructed on some initial data samples. The global optimum is approached by iteratively maximizing a so-called acquisition function, that balances the exploration and exploitation effect of the search. The performance of such an algorithm is largely affected by the choice of the acquisition function. Inspired by the usage of higher moments from the Gaussian process model, it is proposed to construct a novel acquisition function based on the moment-generating function (MGF) of the improvement, which is the stochastic gain over the current best fitness value by sampling at an unknown point. This MGF-based acquisition function takes all the higher moments into account and introduces an additional real-valued parameter to control the trade-off between exploration and exploitation. The motivation, rationale and closed-form expression of the proposed function are discussed in detail. In addition, we also illustrate its advantage over other acquisition functions, especially the so-called generalized expected improvement.

The vehicle routing problem is a classical combinatorial optimization problem. This work is about a variant of the vehicle routing problem with dynamically changing orders and time windows. In real-world applications often the demands change during operation time. New orders occur and others are canceled. In this case new schedules need to be generated on-the-fly. Online optimization algorithms for dynamical vehicle routing address this problem but so far they do not consider time windows. Moreover, to match the scenarios found in real-world problems adaptations of benchmarks are required. In this paper, a practical problem is modeled based on the procedure of daily routing of a delivery company. New orders by customers are introduced dynamically during the working day and need to be integrated into the schedule. A multiple ant colony algorithm combined with powerful local search procedures is proposed to solve the dynamic vehicle routing problem with time windows. The performance is tested on a new benchmark based on simulations of a working day. The problems are taken from Solomon's benchmarks but a certain percentage of the orders are only revealed to the algorithm during operation time. Different versions of the MACS algorithm are tested and a high performing variant is identified. Finally, the algorithm is tested in situ: In a field study, the algorithm schedules a fleet of cars for a surveillance company. We compare the performance of the algorithm to that of the procedure used by the company and we summarize insights gained from the implementation of the real-world study. The results show that the multiple ant colony algorithm can get a much better solution on the academic benchmark problem and also can be integrated in a real-world environment.

Outlier detection in high-dimensional data is a challenging yet important task, as it has applications in, e.g., fraud detection and quality control. State-of-the-art density-based algorithms perform well because they 1) take the local neighbourhoods of data points into account and 2) consider feature subspaces. In highly complex and high-dimensional data, however, existing methods are likely to overlook important outliers because they do not explicitly take into account that the data is often a mixture distribution of multiple components. We therefore introduce GLOSS, an algorithm that performs local subspace outlier detection using global neighbourhoods. Experiments on synthetic data demonstrate that GLOSS more accurately detects local outliers in mixed data than its competitors. Moreover, experiments on real-world data show that our approach identifies relevant outliers overlooked by existing methods, confirming that one should keep an eye on the global perspective even when doing local outlier detection.

Real-world optimization problems are often subject to many constraints. Often, as the volume of the feasible space gets smaller, the problems become more complex. The zero volume of feasible spaces for optimization problems with equality constraints makes them challenging. In this paper, we present an equality constraint handling approach embedded for the first time in a surrogate-assisted optimizer (SACOBRA). The proposed technique starts with an expanded feasible area which gradually shrinks to a volume close to zero. Several well-studied constrained test problems are used as benchmarks and the promising results in terms of efficiency and accuracy are compared with other state-of-the-art algorithms.

Kriging-based Global optimization has been proposed and extensively used for solving black-box optimization problems with expensive function evaluations. The performance of such algorithm relies heavily on the effectiveness of the infill criterion that is used to decide which point to evaluate next. Two common infill criteria are, the probability of improvement (PI) and the expected improvement (EI). The PI results in solutions that have a low risk of failure, that is the solution is not improved in the next round. However the expected gain can be small. EI has a higher risk of failure but maximizes expected gain. This paper views the maximization of PI and EI as a bi-objective optimization problem and suggest a fast and precise gradient-based hypervolume ascent algorithm to compute the Pareto front. The computed Pareto front can be beneficial in different scenarios: Firstly, it can be used for decision making in experimental optimization, when a human decision maker has to decide between a `low risk, low gain' or a `high risk, high gain' strategy. Another application is to use different points on the Pareto front in a multi-point efficient global optimization strategy to balance between exploitation and exploration. The algorithm is validated on two objective functions and compared to other well-known multi-objective optimization algorithms. As a side result we also analyze the landscape of the PI and EI infill criterion and provide closed-form gradient expressions of them.

In optimization with expensive black box evaluations, the expected improvement algorithm (also called efficient global optimization) is a commonly applied method. It uses Gaussian Processes (or Kriging) to build a model of the objective function and uses the expected improvement as an infill criterion, taking into account both — predictive mean and variance. It has been generalized to multi-objective optimization using the expected hypervolume improvement, which measures the expected gain in the hypervolume indicator of a Pareto front approximation. However, this criterion assumes an unbounded objective space even if it is often known a-priori that the objective function values are within a prescribed range, e.g., lower bounded by zero. To take advantage of such a-priori knowledge, this paper introduces the truncated expected hypervolume improvement and a multiobjective efficient global optimization method that is based on it. In this paper it is shown how to compute the truncated expected hypervolume improvement exactly and efficiently. Then it is tested as an infill criterion in efficient global optimization. It is shown that it can effectively make use of a-priori knowledge and achieve better results in cases where such knowledge is given. The usefulness of the new approach is demonstrated in benchmark examples and applications from robust PID (proportionalintegral-derivative) controller optimization. The empirical studies in this paper are confined to the bi-objective case.

Kriging or Gaussian Process Regression has been successfully applied in many fields. One of the major bottlenecks of Kriging is the complexity in both processing time (cubic) and memory (quadratic) in the number of data points. To overcome these limitations, a variety of approximation algorithms have been proposed. One of these approximation algorithms is Optimally Weighted Cluster Kriging (OWCK). In this paper, OWCK is extended and enhanced by the use of fuzzy clustering methods in order to increase the accuracy. Several options are proposed and evaluated against both the original OWCK and a variety of other Kriging approximation algorithms.

In this contribution a derivative-free memetic algorithm (MA) for the design optimization of chemical processes is introduced. Design optimization problems are characterized by nonlinear cost functions and highly constrained and multi-modal search spaces. The MA is a combination of an evolution strategy that addresses the global optimization of discrete and continuous design decisions and a derivative-free optimization method (DFO) that performs a local optimization of the continuous sub-problems that remain after fixing the discrete decisions. The MA calls a process simulation software to simulate the design alternatives, i.e. the evaluation of the objective and the constraints is a black box. In this contribution, the focus lies on the selection of a suitable DFO solver for efficiently solving the continuous constrained sub-problems. Based on latin-hypercube samplings of sub-problems of two instances of a real-world case study, surrogate models for the objective and the constraints were generated. A set of DFO methods was tested and compared on the surrogate models. The method that showed the best performance was coupled to the MA which was then applied to the real-world case study.

Truck Loading Optimization problems from practice tend to involve a lot of constraints. In automatically solving these problems, a rule set has to be compiled that governs the generation of valid solutions. This rule set is either defined manually, or distilled automatically by analyzing solutions of human planners. This paper describes the extension of an existing optimization approach with statistics from man-made solutions through a so-called informed mutation operator. Solutions are scored on quality and on the percentage of violation, i.e., a combination of stacked boxes that did not occur in a separate set of man-made solutions. Applying the informed mutation operator decreases the average violation percentage in solutions, as well as achieving improved solution quality through fitting more boxes into a container.

Multi-objective optimization problems with many objective functions (3> 3) are difficult to solve. This is because the time complexity of optimization algorithms often grows fast with the number of objective functions and the results of many objective optimization algorithms (Pareto front approximation) require large memory and their interpretation can be difficult for the decision maker. It is therefore attractive to reduce complexity of these problems. This can be achieved by decomposing them into a set of independent lower dimensional subproblems, or by aggregating some objective functions into a single objective function. This work introduces a new approach for decomposition and aggregation based on techniques from social network analysis. The key idea is to interpret an objective function as a node (agent) in a social network, and arcs between nodes indicate relationships: Negatively weighted arcs stand for conflicting objectives, zero weighted arcs for independent objectives, and positively weighted arcs for objectives that support each other. Using well-known algorithms BGLL for community detection we show that — given certain preconditions — it is possible to decompose a many objective optimization problem to a set of lower dimensional multi-objective optimization problems. This makes it easier to solve the problem and interpret the resulting trade-off (hyper-)surfaces. In the paper, after introducing the idea of Communtiy Detection for Many-Objective Optimization (CoDeMO), we lay out an interactive workflow and test it on simple, scalable many-objective optimization problems-variants of facility location problems — and thereby provide a prove-of-concept study.

Most research into the Container Loading problem has been done on theoretical problem sets and while taking one or two constraints into account. In this paper we discuss the successful implementation of a self-adaptive Genetic Algorithm applying only mutation, with a variable mutation rate. This is applied to a real-world problem with actual problem instances from industry. We introduce an abstract, indirect representation for the considered loadings together with two mutation strategies. Solutions of these different strategies are compared with each other, a static mutation rate GA, and with solutions created by human planners as used in industry, for a set of over 500 realworld problem instances. Furthermore, we examine how our automated results compare to those generated by experienced human planners, showing that they are valid loadings and match fitness values.

In this paper, we propose an ant based solver to solve dynamic vehicle routing problem with time windows and multiple priorities (DVRPTWMP). In this problem, a fleet of vehicles will service a number of customers with time windows. But part of customers are unknown and revealed dynamically during the execution of the routes. More specifically, customers have different priority levels. Customers with higher priority should have a higher quality of service. The quality of service is measured by the sum of the expected delay time between the arrival time and the earliest available beginning service time of all customers. The goal is to minimize the traveling distance while minimizing the total delay time of customers. First, a new benchmark is generated for DVRPTWMP based on van Veen's benchmark which is a dynamical extension of Solomon's 100 customers benchmark. Then, the ant based solver is introduced and two strategies based on ant colony algorithm are proposed to deal with priorities of customers. One is servicing high priority level customers immediately using the nearest vehicle. The other is giving a penalty to the delay time, combining the penalty and traveling distance and minimizing them together. Finally, the results show that the second strategy performs better than the first one.

This paper presents and analyses an engineered expected hypervolume improvement (EHVI) algorithm for solving the problem of PID parameter tuning and the optimization problem of controlling the substrate feed of a biogas plant. The EHVI is the expected value of the increment of the hypervolume indicator given a Pareto front approximation and a predictive multivariate Gaussian distribution of a new point. To solve this problem, S-metric selection-based efficient global optimization (SMS-EGO), EHVI based efficient global optimization (EHVIEGO) and SMS-EMOA are used and compared in both the PID parameter tuning problem and for biogas plant feed optimization. The results of the experiments show that surrogate model based algorithms perform better than SMS-EMOA, and the performance of EHVI-EGO is slightly better than SMS-EGO.

Complex networks rely on their structural robustness for their function and performance. Considering that redundancy backup is frequently used to enhance the robustness of complex networks, we try to find better redundancy backup strategy by using optimization methods. In this paper, our contributions are twofold. First, we prove that natural connectivity is suitable for measuring network robustness. Second, a robustness optimization algorithm is proposed based on GA, while it is different from traditional GA. The method of coding, crossover and mutation operations are all improved in this research. Extensive experiments on real-world datasets demonstrate that the effectiveness of our methods is better than the classical rich-rich redundancy backup strategy.

Nowadays, constraints play an important role in industry, because most industrial optimization tasks underly several restrictions. Finding good solutions for a particular problem with respect to all constraint functions can be expensive, especially when the dimensionality of the search space is large and many constraint functions are involved. Unfortunately function evaluations in industrial optimization are heavily limited, because often expensive simulations must be conducted. For such high-dimensional optimization tasks, the constraint optimization algorithm COBRA was proposed, making use of surrogate modeling for both the objective and the constraint functions. In this paper we present a new mechanism for COBRA to repair infill solutions with slightly violated constraints. The repair mechanism is based on gradient descent on surrogates of the constraint functions and aims at finding nearby feasible solutions. We test the repair mechanism on a real-world problem from the automotive industry and on other synthetic test cases. It is shown in this paper that with the integration of the repair method, the percentage of infeasible solutions is significantly reduced, leading to faster convergence and better final results.

The receiver operating characteristic (ROC) is commonly used to analyze the performance of classifiers in data mining. An important topic in ROC analysis is the ROC convex hull (ROCCH), which is the least convex majorant (LCM) of the empirical ROC curve and covers potential optima for a given set of classifiers. ROCCH maximization problems have been taken as multiobjective optimization problem (MOPs) in some previous work. However, the special characteristics of ROCCH maximization problem makes it different from traditional MOPs. In this paper, the difference will be discussed in detail and a new convex hull-based multiobjective genetic programming (CH-MOGP) is proposed to solve ROCCH maximization problems. Specifically, convex hull-based without redundancy sorting (CWR-sorting) is introduced, which is an indicator-based selection scheme that aims to maximize the area under the convex hull. A novel selection procedure is also proposed based on the proposed sorting scheme. It is hypothesized that by using a tailored indicator-based selection, CH-MOGP becomes more efficient for ROC convex hull approximation than algorithms that compute all Pareto optimal points. Empirical studies are conducted to compare CH-MOGP to both existing machine learning approaches and multiobjective genetic programming (MOGP) methods with classical selection schemes. Experimental results show that CH-MOGP outperforms the other approaches significantly.

Academic education of professional processes is challenged by a necessary balance of practical activities with academic reflection. In this paper we address this issue by discussing our experiences with teaching software engineering practices and their continuous improvement. By designing a graduate course we embed an intensive coaching routine based upon agile practices with research activities to leverage knowledge of students and coaches. As a concrete example of an embedded research project we conduct an experiment on the impact of two different meeting routines on the teams satisfaction with information exchange. Our results show that the intensive coaching in individual teams is shorter in nature and more appealing to the students. Our findings suggest that software engineering education can benefit from the notion of team routines and process improvement practices contributing to maturity of students and educators.

Measures of novelty and interestingness are frequently encountered in the context of developmental robotics, being derived from human psychology. This work addresses these measures from the viewpoint of enhancing design-space exploration in black-box optimization. We provide a unifying notational and naming scheme with the intent of facilitating comparison, implementation, and application in the domain of design optimization. Initial analysis shows a promising interestingness measure for being tried on real-world design problems.

Evolution strategies (ESs) are powerful probabilistic search and optimization algorithms gleaned from biological evolution theory. They have been successfully applied to a wide range of real world applications. The modern ESs are mainly designed for solving continuous parameter optimization problems. Their ability to adapt the parameters of the multivariate normal distribution used for mutation during the optimization run makes them well suited for this domain. In this article we describe and study mixed integer evolution strategies (MIES), which are natural extensions of ES for mixed integer optimization problems. MIES can deal with parameter vectors consisting not only of continuous variables but also with nominal discrete and integer variables. Following the design principles of the canonical evolution strategies, they use specialized mutation operators tailored for the aforementioned mixed parameter classes. For each type of variable, the choice of mutation operators is governed by a natural metric for this variable type, maximal entropy, and symmetry considerations. All distributions used for mutation can be controlled in their shape by means of scaling parameters, allowing self-adaptation to be implemented. After introducing and motivating the conceptual design of the MIES, we study the optimality of the self-adaptation of step sizes and mutation rates on a generalized (weighted) sphere model. Moreover, we prove global convergence of the MIES on a very general class of problems. The remainder of the article is devoted to performance studies on artificial landscapes (barrier functions and mixed integer NK landscapes), and a case study in the optimization of medical image analysis systems. In addition, we show that with proper constraint handling techniques, MIES can also be applied to classical mixed integer nonlinear programming problems.

The Capacitated Vehicle Routing Problem (CVRP) is a widely studied, NP-hard problem with many real-world applications. Exact approaches are infeasible for solving large problem instances, due to the superpolynomial time complexity. Therefore, most solution approaches over the years have been metaheuristics, such as the Genetic Algorithm (GA). This paper presents a Hybrid GA, which incorporates problem-specific heuristics and domain knowledge into the algorithm. This causes the parameter settings to behave slightly different from regular GAs. To take care of the parameterization, an additional GA is used, acting on the Hybrid GA. This will be referred to as the Meta-GA. In addition to solving all known problems optimally or within 1% of the optimum, it manages to find a new best result within research literature for M-n200-k16, one of the largest problem instances.

Identification and validation of material models for forming simulations to match experimental data is a key requirement of complex forming applications in the automotive industry. Besides the fact that these models need more and relatively expensive material data input, a problem is still the reliable fit of all necessary parameters. For the steel grade DX54, an interaction of strain rate and yield locus has been identified, finally leading to different material model calibrations. An even better accuracy of feasibility studies is promoted by using advanced yield locus models for forming simulations. With a new set of specially designed experiments in combination with evolutionary strategies, inverse material parameter identification is realized through minimization of a nonlinear error function. This approach defines a new and powerful method for selection and validation of the adequate material model for industrial simulation.

This paper discusses the adoption of self-adaptation for Evolutionary Algorithms operating in binary spaces using a direct encoding of the mutation rate. In particular, it focuses on the log-normal update rule for adapting the mutation rate, incorporated in a (mu, lambda)-strategy. Although it is well known that this update rule requires a lower boundary of the mutation rate to prevent it from collapsing to zero, the naive approach of enforcing a fixed lower boundary has undesirable side-effects. This paper studies the dynamics of the fixed lower boundary approach in depth and proposes a simple alternative for dealing with the lower boundary issue.

Algorithms that search for robust optima often evaluate the effective fitness (robust fitness) based on stochastic approximation schemes. In this setting, finding robust optima can be recast as an optimization problem with an/a uncertain/noisy objective function. This paper studies whether state-of-the-art uncertainty handling techniques, proposed in the context of optimizing noisy objective functions, can be applied for finding robust optima. In this paper, the UH-CMA-ES is modified to handle approximations of the effective fitness. This modified approach, named RO-UH-CMA-ES, is evaluated empirically and compared to other schemes that aim to find robust optima. The experiments on multiple benchmark problems show that the RO-UH-CMA-ES yields comparable results for multi-modal problems and it outperforms other schemes on unimodal problems.

This paper presents a study for using Kriging metamodeling in combination with Covariance Matrix Adaptation Evolution Strategies (CMA-ES) to find robust solutions. A general, archive based, framework is proposed for integrating Kriging within CMA-ES, including a method to utilize the covariance matrix of the CMA-ES in a straightforward way to improve the accuracy of the Kriging predictions without introducing much additional computational cost. Moreover, it adopts an elegant way to select appropriate archive points for building a local metamodel. The study shows that this Kriging metamodeling scheme for finding robust solutions outperforms common, straightforward approaches and is very useful when there is a limited budget of function evaluations. Though using the covariance matrix can improve the prediction quality, it has no significant effect on the overall quality of the optimization results.

This paper reports about the application of a new variant of multiobjective Mixed-Integer Evolution Strategy to a warehouse design optimization problem. The algorithm is able to deal with real-valued, integer, and discrete nominal input variables in a multiobjective problem, and utilizes a multiobjective selection procedure based on either crowding distance or hypervolume contribution (also called S metric). The warehouse design optimization problem investigated in this study is represented by a warehouse simulator (provided by our collaboration partner) which calculates four warehouse performance measures: total handling time, crate fill rate, order latency, and investment cost. Two of those are treated as objectives, while the other two are represented as constraints. As demonstrated by the results, the algorithm generates solutions which cover the whole Pareto front, as opposed to the expert-generated solutions. Moreover, the algorithm is able to find solutions which improve on the expert-generated solutions with respect to both objectives.

While the motivation and usefulness of niching methods is beyond doubt, the relaxation of assumptions and limitations concerning the hypothetical search landscape is much needed if niching is to be valid in a broader range of applications. Upon the introduction of radii-based niching methods with derandomized evolution strategies (ES), the purpose of this study is to address the so-called niche radius problem. A new concept of an adaptive individual niche radius is applied to niching with the covariance matrix adaptation evolution strategy (CMA-ES). Two approaches are considered. The first approach couples the radius to the step size mechanism, while the second approach employs the Mahalanobis distance metric with the covariance matrix mechanism for the distance calculation, for obtaining niches with more complex geometrical shapes. The proposed approaches are described in detail, and then tested on high-dimensional artificial landscapes at several levels of difficulty. They are shown to be robust and to achieve satisfying results.

There exist several applications of multi-objective evolutionary algorithms for drug design, however, a common drawback in recent approaches is that the diversity of resulting molecule populations is relatively low. This paper seeks to overcome this problem by introducing niching as a technique to enhance search space diversity. A single population approach with dynamic niche identification is studied in the application domain. In order to apply niching in molecular spaces a metric for measuring the dissimilarity of molecules will be introduced. The approach will be validated in case studies and compared with results of an NSGA-II algorithm without niching in the search space.

This paper discusses mixed integer evolution strategies (MIES) assisted by metamodels based on radial basis function networks (RBFN). The goal is to make MIES more suitable for optimization with time consuming evaluation functions.

This paper introduces the optimization of a quantum control application, the so-called molecular alignment problem, subject to a dynamic environment. Given the relative simplicity of optimized pulse-shapes in the low-intensity variant of the problem, versus the high complexity of the optimized pulse-shapes in the high-intensity case, a dynamic-intensity environment is simulated in a noise-free calculation. Specific evolution strategies, natural candidates for optimization in dynamic environments, are applied to this task. The calculations reveal the evolution of the pulse-shapes and their underlying evolving structures, that allow a complete physical interpretation. The combination of an optimization in a dynamic environment with the examination of the intermediate optimized solutions offers a sharper physics view of the problem, and accomplishes a fruitful interdisciplinary study.

Self-adaptation is used a lot in Evolutionary Strategies and with great success, yet for some reason it is not the mutation adaptation of choice for Genetic Algorithms. This poster describes how a self-adaptive mutation rate was used in a Genetic Algorithms to inverse design behavioral rules for a Cellular Automata. The unique characteristics of this search space gave rise to some interesting convergence behavior that might have implications for using self-adaptive mutation rates in other Genetic Algorithm applications and might clarify why self-adaptation in Genetic Algorithms is less successful than in Evolutionary Strategies.

This paper presents an evolutionary algorithm for the automated design of molecules that could be used as drugs. It is designed to provide the medicinal chemist with a number of candidate molecules that comply to pre-defined properties. These candidate molecules can be promising for further evaluation.

Genetic Algorithms (GAs) are historically the most commonly used optimization method in Quantum Control (QC) experiments. We transfer specific Derandomized Evolution Strategies (DES) that have performed well on noise-free theoretical Quantum Control calculations, including the Covariance Matrix Adaptation (CMA-ES) algorithm, into the noisy environment of Quantum Control experiments. We study the performance of these DES variants in laboratory experiments, and reveal the underlying strategy dynamics of first- versus second-order landscape information.

Existing niching techniques commonly use the Euclidean distance metric in the decision space for the classification of feasible solutions to the niches under formation. This approach is likely to encounter problems in high-dimensional landscapes with non-isotropic basins of attraction. Here we consider niching with the covariance matrix adaptation evolution strategy (CMA-ES), and introduce the Mahalanobis distance metric into the niching mechanism, aiming to allow a more accurate spatial classification, based on the ellipsoids of the distribution, rather than hyper-spheres of the Euclidean metric. This is tested with the CMA-(+) routines, and compared to two niching frameworks - fixed niche radius as well as self- adaptive niche radius, which is based on the coupling to the step-size. The performance of the different variants is evaluated on a suite of theoretical test-functions. We thus present here the Mahalanobis-assisted CMA-niching as a state-of-the-art niching technique within evolution strategies (ES), and propose it as a solution to the so-called niche radius problem.

This study introduces the multi-criteria approach to the optimization of dynamic molecular alignment by shaped femtosecond laser pulses, which has been considered so far only as a single-criterion problem. The paper applies advanced Pareto front approximation algorithms to this challenging real-world, high-dimensional, and computationally expensive problem, working with low-dimensional parameterizations of the electric field. Standard approaches (NSGA-II) and their metamodel-assisted extensions based on Kriging, are applied to this optimization task and compared among each other. The study confirms the conflicting nature of the objectives. Interesting features of the problem domain, such as the geometry of the Pareto front are revealed. Furthermore, metamodel-assistance, in particular pre-screening with the Kriging-based expected improvement criterion, proves to be a valuable ingredient for improving the numerical results.

A survey of niching algorithms, based on 5 variants of derandomized Evolution Strategies (ES), is introduced. This set of niching algorithms, ranging from the very first derandomized approach to self-adaptation of ES to the sophisticated (1 + , λ) Covariance Matrix Adaptation (CMA), is applied to multimodal continuous theoretical test functions, of different levels of difficulty and various dimensions, and compared with the MPR performance analysis tool. While characterizing the performance of the different derandomized variants in the context of niching, some conclusions concerning the niching formation process of the different mechanisms are drawn, and the hypothesis of a tradeoff between learning time and niching acceleration is numerically confirmed. Niching with (1+λ)-CMA core mechanism is shown to experimentally outperform all the other variants. Some theoretical arguments supporting the advantage of a plus-strategy for niching are discussed.

The Second Harmonic Generation (SHG), a process that turns out to be a good test case in the physics lab, can also be considered as a fairly simple theoretical test function for global optimization. Despite its symmetry properties, that will be derived here analytically, it seems to capture the complexity of the Fourier transform between the decision space to the evaluation space, and by that to challenge optimization routines. And indeed, counter-intuitively to some extent, locating its global maximum seems to be not an easy task for Evolutionary Algorithms (EAs).

We consider the numerical optimization of dynamic molecular alignment by shaped femtosecond pulses, and study the scalability of the electric field subject to optimization by Evolution Strategies.The trade-off between fine-tuning of the electric field versus the evolutionary optimization feasibility is investigated.

This paper introduces the optimization of dynamic molecular alignment by shaped femtosecond laser pulses, and analyzes the application of various Evolutionary Algorithms to this challenging real-life high-dimensional physics problem. With an expensive simulator evaluation of 35 seconds, standard evolutionary approaches based on low-dimensional parameterizations of the electric field are applied to the task, compared among each other, and shown to be clearly inferior with respect to other methods. This numerical phase provides new insights into the problem, and is meant to be followed by a lab phase.

This paper presents a new parameterization method for the Evolution Strategies (ES) field, and its application to a challenging real-life high-dimensional Physics optimization problem, namely Femtosecond Laser Pulse Shaping. The so-called Complete-Basis-Functions Parameterization method (CBFP), to be introduced here for the first time, is developed for tackling efficiently the given laser optimization task, but nevertheless is a general method that can be used for learning any n-variables functions. The emphasis is on dimensionality reduction of the search space and the speeding-up of the convergence process respectively. This is achieved by learning the target function by using complete-basis functions as building blocks in an evolutionary search. The method is shown to boost the learning process of the given laser problem, and to yield highly satisfying results.

Evolutionary algorithms (EAs) have the tendency to converge quickly into a single solution in the search space. However, many complex search problems require the identification and maintenance of multiple solutions. Niching methods are the extension of EAs to address this issue. In our study, we propose an evolution strategy (ES) niching method, based on the covariance matrix adaptation (CMA) mechanism. We analyze our algorithm, introduce an experimental setup, and compare its performance with a previous ES niching method, known as the ES dynamic niching algorithm. In our comparison we introduce for the first time a new analytical tool for niching analysis, and in particular the early niching formation process. Based on successful data fit, we propose the well-known logistic model to describe our experimental results.

Cellular automata are used in many fields to generate a global behavior with local rules. Finding the rules that display a desired behavior can be a hard task especially in real world problems. This paper proposes an improved approach to generate these transition rules for multi dimensional cellular automata using a genetic algorithm, thus giving a generic way to evolve global behavior with local rules, thereby mimicking nature. Three different problems are solved using multi dimensional topologies of cellular automata to show robustness, flexibility and potential. The results suggest that using multiple dimensions makes it easier to evolve desired behavior and that combining genetic algorithms with multi dimensional cellular automata is a very powerful way to evolve very diverse behavior and has great potential for real world problems.

EAs have the tendency to converge quickly into a single solution. Niching methods, the extension of EAs to address this issue, have been investigated up to date mainly within the field of Genetic Algorithms (GAs). In our study we investigate the basis for niching methods within Evolution Strategies (ES), and propose the first ES niching method. Results show that this method can reliably find and maintain multiple niches even for high-dimensional problems.

Neural networks and Kriging method are compared for constructing fitness approximation models in evolutionary optimization algorithms. The two models are applied in an identical framework to the optimization of a number of well known test functions. In addition, two different ways of training the approximators are evaluated: in one setting the models are built off-line using data from previous optimization runs and in the other setting the models are built online from the data available from the current optimization.

Methods are developed to numerically analyze an evolutionary algorithm (EA) that applies mutation and selection on a bit-string representation to find the optimum for a bimodal unitation function called a trap function. This research bridges part of the gap between the existing convergence velocity analysis of strictly unimodal functions and global convergence results assuming the limit of infinite time. As a main result of this analysis, a new so-called (1 : /spl lambda/)-EA is proposed, which generates offspring using individual mutation rates p/sub i/. While a more traditional EA using only one mutation rate is not able to find the global optimum of the trap function within an acceptable (nonexponential) time, our numerical investigations provide evidence that the new algorithm overcomes these limitations. The analysis tools used for the analysis, based on absorbing Markov chains and the calculation of transition probabilities, are demonstrated to provide an intuitive and useful method for investigating the capabilities of EAs to bridge the gap between a local and a global optimum in bimodal search spaces.

The starting point for the analysis and experiments presented in this paper is a simplified elevator control problem, called 'S-ring'. As in many other real-world optimization problems, the exact fitness function evaluation is disturbed by noise. Evolution strategies (ES) can generally cope with noisy fitness function values. It has been proposed that the 'plus'-strategy can find better solutions by keeping over-valued function values, thus preventing inferior offspring with fitness inflated by noise from being accepted. The 'plus'-strategy builds an implicit barrier around the current best population. We propose to make this barrier building process explicit and to employ a threshold value /spl tau/ to be used in a selection operator for noisy fitness functions. 'Thresholding' accepts a new individual if its apparent fitness is better than that of the parent by at least the margin /spl tau/. First analytical investigations and empirical results from tests on the sphere-model and 'S-ring' are presented.

This paper describes a project funded by the European Commission which seeks to provide the technology and software infrastructure necessary to support the next generation of evolving infohabitants in a way that makes that infrastructure universal, open and scalable. The Distributed Resource Evolutionary Algorithm Machine (DREAM) will use existing hardware infrastructure in a more efficient manner, by utilising otherwise unused CPU time. It will allow infohabitants to co-operate, communicate, negotiate and trade; and emergent behaviour is expected to result. It is expected that there will be an emergent economy that results from the provision and use of CPU cycles by infohabitants and their owners. The DREAM infrastructure will be evaluated with new work on distributed data mining, distributed scheduling and the modelling of economic and social behaviour.

In this paper two different generalizations of intermediate recombination in evolution strategies are investigated. Both generalizations allow for recombining an arbitrary number of /spl rho/ parents. However, the so-called /spl rho///spl rho/-mechanism averages all /spl rho/ parents, while the so-called /spl rho//2-mechanism repeatedly (for each object variable anew) selects two out of /spl rho/ parents and averages the corresponding object variables to create an offspring individual. Results presented for the spherical function demonstrate that these two operators can cause a significantly different behavior concerning the convergence velocity of the algorithm. Both operators are applied to a number of different objective functions (including separable and non-separable, unimodal and multimodal, regular and irregular topologies), and the impact of the number of parents /spl rho/ is investigated. The results illustrate that important differences in the results are not consistent with the canonical topology classification of objective functions, but can be explained to some extent by the "genetic repair" hypothesis of Beyer in combination with a reasoning about the success region.

The potential for cross-fertilization between the fields of DNA based computing and evolutionary computation is outlined both from a principal point of view and by means of an experimental investigation concerning the NP-hard maximum clique problem. A simple evolutionary approach to maximum clique is introduced and the hypothesis is tested whether the increase in population size possible by realizing evolutionary computation with DNA yields the expected improvement in solution quality. Results obtained for a limited range of population sizes up to 10/sup 4/ indicate that the hypothesis holds for about two-third of the investigated problem instances (which were taken from the DIMACS library).

The principle of self-adaptation in evolutionary algorithms is an important mechanism for controlling the strategy parameters of such algorithms by evolving parameter values in analogy with the usual evolution of object variables. To facilitate evolu

Robustness is an important requirement for almost all kinds of products. This article shows how evolutionary algorithms can be applied for robust design based on the approach of Taguchi. To achieve a better understanding of the consequences of this approach, we first present some analytical results gained from a toy problem. As a nontrivial industrial application we consider the design of multilayer optical coatings (MOCs) most frequently used for optical filters. An evolutionary algorithm based on a parallel diffusion model and extended for mixed-integer optimization was able to compete with or even outperform traditional methods of robust MOC design. With respect to chromaticity, the MOC designs found by the evolutionary algorithm are substantially more robust to parameter variations than a reference design and therefore perform much better in the average case. In most cases, however, this advantage has to be paid for by a reduction in the average reflectance. The robust design approach outlined in this paper should be easily adopted to other application domains.

An extension of evolution strategies to multiparent recombination involving a variable number [symbol: see text] of parents to create an offspring individual is proposed. The extension is experimentally evaluated on a test suite of functions differing in their modality and separability and the regular/irregular arrangement of their local optima. Multiparent diagonal crossover and uniform scanning crossover and a multiparent version of intermediary recombination are considered in the experiments. The performance of the algorithm is observed to depend on the particular combination of recombination operator and objective function. In most of the cases a significant increase in performance is observed as the number of parents increases. However, there might also be no significant impact of recombination at all, and for one of the unimodal objective functions, the performance is observed to deteriorate over the course of evolution for certain choices of the recombination operator and the number of parents. Additional experiments with a skewed initialization of the population clarify that intermediary recombination does not cause a search bias toward the origin of the coordinate system in the case of domains of variables that are symmetric around zero.

Evolutionary computation has started to receive significant attention during the last decade, although the origins can be traced back to the late 1950's. This article surveys the history as well as the current state of this rapidly growing field. We describe the purpose, the general structure, and the working principles of different approaches, including genetic algorithms (GA) (with links to genetic programming (GP) and classifier systems (CS)), evolution strategies (ES), and evolutionary programming (EP) by analysis and comparison of their most important constituents (i.e. representations, variation operators, reproduction, and selection mechanism). Finally, we give a brief overview on the manifold of application domains, although this necessarily must remain incomplete.

The fifth event in the series of International Conferences on Genetic Algorithms (ICGA), which have been held in the United States each odd year since 1985, took place in July 1993 a t the University of Illinois a t Urbana-Champaign. It was a very well organized meeting. The steadily growing interest in the research field of evolutionary computation, of which genetic algorithms are a part, was well documented by the large number of par-ticipants (more than 300). Consequently, the conference proceedings are rather impressive, collecting 1 18 contributions (83 full papers and 3 5 one-page summaries) on 665 pages, and the Morgan Kaufmann softcover edition is reasonably priced. A classification of the papers according to the affiliation of the first author reveals that more than half of the contributions (63) originate from the United States, followed by two strong groups formed by the United Kmgdom (14) and Japan (13). With the exception of Germany (7), the remaining papers are split over a variety of countries that are represented by fewer than four papers each. Though this small exercise in counting confirms the international character of the conference, the scientific content of the contributions unfortunately does not represent the complete scope of evolutionary computation. The proceedings contain papers related to

Three main streams of evolutionary algorithms (EAs), probabilistic optimization algorithms based on the model of natural evolution, are compared in this article: evolution strategies (ESs), evolutionary programming (EP), and genetic algorithms (GAs). The comparison is performed with respect to certain characteristic components of EAs: the representation scheme of object variables, mutation, recombination, and the selection operator. Furthermore, each algorithm is formulated in a high-level notation as an instance of the general, unifying basic algorithm, and the fundamental theoretical results on the algorithms are presented. Finally, after presenting experimental results for three test functions representing a unimodal and a multimodal case as well as a step function with discontinuities, similarities and differences of the algorithms are elaborated, and some hints to open research questions are sketched.