Determining the number of algorithm runs is a critical aspect of experimental design, as it directly influences the experiment's duration and the reliability of its outcomes. This paper introduces an empirical approach to estimating the required number of runs per problem instance for accurate estimation of the performance of the continuous single-objective stochastic optimization algorithm. The method leverages probability theory, incorporating a robustness check to identify significant imbalances in the data distribution relative to the mean, and dynamically adjusts the number of runs during execution as an online approach.

Characterization of the optimization problem is a crucial task in many recent optimization research topics (e.g., explainable algorithm performance assessment, and automated algorithm selection and configuration). The state-of-the-art approaches use exploratory landscape analysis to represent the optimization problem, where for each one, a set of features is extracted using a set of candidate solutions sampled by a sampling strategy over the whole decision space. This paper proposes a novel representation of continuous optimization problems by encoding the information found in the interaction between an algorithm and an optimization problem. The new problem representation is learned using the information from the states/positions in the optimization run trajectory (i.e., the candidate solutions visited by the algorithm). With the novel representation, the problem can be characterized dynamically during the optimization run, instead of using a set of candidate solutions from the whole decision space that have never been observed by the algorithm. The novel optimization problem representation is called Opt2Vec and uses an autoencoder type of neural network to encode the information found in the interaction between an optimization algorithm and optimization problem into an embedded subspace. The Opt2Vec representation efficiency is shown by enabling different optimization problems to be successfully identified using only the information obtained from the optimization run trajectory.

Leave-one-problem-out (LOPO) performance prediction requires machine learning (ML) models to extrapolate algorithms' performance from a set of training problems to a previously unseen problem. LOPO is a very challenging task even for state-of-the-art approaches. Models that work well in the easier leave-one-instance-out scenario often fail to generalize well to the LOPO setting. To address the LOPO problem, recent work suggested enriching standard random forest (RF) performance regression models with a weighted average of algorithms' performance on training problems that are considered similar to a test problem. More precisely, in this RF+clust approach, the weights are chosen proportionally to the distances of the problems in some feature space. Here in this work, we extend the RF+clust approach by adjusting the distance-based weights with the importance of the features for performance regression. That is, instead of considering cosine distance in the feature space, we consider a weighted distance measure, with weights depending on the relevance of the feature for the regression model. Our empirical evaluation of the modified RF+clust approach on the CEC 2014 benchmark suite confirms its advantages over the naive distance measure. However, we also observe room for improvement, in particular with respect to more expressive feature portfolios.

The application of machine learning (ML) models to the analysis of optimization algorithms requires the representation of optimization problems using numerical features. These features can be used as input for ML models that are trained to select or to configure a suitable algorithm for the problem at hand. Since in pure black-box optimization information about the problem instance can only be obtained through function evaluation, a common approach is to dedicate some function evaluations for feature extraction, e.g., using random sampling. This approach has two key downsides: (1) It reduces the budget left for the actual optimization phase, and (2) it neglects valuable information that could be obtained from a problem-solver interaction.

In black-box optimization, it is essential to understand why an algorithm instance works on a set of problem instances while failing on others and provide explanations of its behavior. We propose a methodology for formulating an algorithm instance footprint that consists of a set of problem instances that are easy to be solved and a set of problem instances that are difficult to be solved, for an algorithm instance. This behavior of the algorithm instance is further linked to the landscape properties of the problem instances to provide explanations of which properties make some problem instances easy or challenging. The proposed methodology uses meta-representations that embed the landscape properties of the problem instances and the performance of the algorithm into the same vector space. These meta-representations are obtained by training a supervised machine learning regression model for algorithm performance prediction and applying model explainability techniques to assess the importance of the landscape features to the performance predictions. Next, deterministic clustering of the meta-representations demonstrates that using them captures algorithm performance across the space and detects regions of poor and good algorithm performance, together with an explanation of which landscape properties are leading to it.

Exploratory Landscape Analysis (ELA) enables the characterization of black-box optimization problem instances in the form of numerical features. Such features can be used to train a Machine Learning (ML) model to automatically predict the performance of an optimization algorithm on a specific problem instance. However, computing ELA features is a time consuming process and relatively expensive. In this paper, we aim to evaluate the usefulness of ELA features and identify features which are the most informative in automated algorithm performance prediction. The goal is to find a subset of features which are sufficient to train a reliable ML model for algorithm performance prediction, with reduced computational costs for calculating the ELA features. We focus on the performance prediction of the Covariance Matrix Adaptation Evolution Strat-egy (CMA-ES) algorithm on the COCO benchmark problems. The results showed that the number of ELA features that lead to a reliable algorithm performance prediction depends on the modular CMA-ES configuration under consideration. However, the set of features that are selected to be useful across different modular CMA-ES configurations are similar.

Exploratory Landscape Analysis is a powerful technique that allows us to gain an understanding of a problem landscape solely by sampling the problem space. It has been successfully used in a number of applications, for example for the task of automatic algorithm selection. However, recent work has shown that Exploratory Landscape Analysis contains some specific weaknesses that its users should be aware of. As the technique is sample based, it has been shown to be sensitive to the choice of sampling strategy. Additionally, many landscape features are not invariant to transformations of the underlying samples which should have no effect on algorithm performance, specifically shifting and scaling. The analysis of the effect of shifting and scaling has so far only been demonstrated on a single problem set and dimensionality. In this paper, we perform a comprehensive analysis of the invariance of Exploratory Landscape Analysis features to these two transformations, by considering different sampling strate-gies, sampling sizes, problem dimensionalities, and benchmark problem sets to determine their individual and combined effect. We show that these factors have very limited influence on the features' invariance when they are considered either individually or combined.

Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks.

Selecting the most suitable algorithm and determining its hyperparameters for a given optimization problem is a challenging task. Accurately predicting how well a certain algorithm could solve the problem is hence desirable. Recent studies in single-objective numerical optimization show that supervised machine learning methods can predict algorithm performance using landscape features extracted from the problem instances.

In this paper, we have proposed a new pipeline for landscape analysis of time-series machine learning datasets that enables us to better understand a benchmarking problem landscape, allows us to select a diverse benchmark datasets portfolio, and reduces the presence of performance assessment bias via bootstrapping evaluation. Combining a large multi-domain representation corpus of time-series specific features and the results of a large empirical study of time-series classification (TSC) benchmark, we showcase the capability of the pipeline to point out issues with non-redundancy and representativeness in the benchmark. By observing discrepancy between the empirical results of the bootstrap evaluation and recently adopted practices in TSC literature when introducing novel methods, we warn on the potentially harmful effects of tuning the methods on certain parts of the landscape (unless this is an explicit and desired goal of the study). Finally, we propose a set of datasets uniformly distributed across the landscape space one should consider when benchmarking novel TSC methods.

Exploratory Landscape Analysis is a methodology for transforming the samples of an optimization problem into numerical descriptors called landscape features. Since Exploratory Landscape Analysis is sample based, recent studies have shown that the choice of the method used to collect the problem samples can have an effect on the calculation of landscape features.In our recent work, we have shown that certain landscape features are not invariant to even simple function transformations such as shifting or scaling. However, the analysis in our previous work was conducted using only Latin Hypercube Sampling. Since we are now aware that the choice of sampling method can affect the calculation of landscape features, this paper expands upon our earlier work by using a variety of different sampling methods. We show that different sampling methods do indeed have an effect on the invariance of landscape features, and present a list of landscape features that are invariant under all of our chosen sampling methods.

Accurately predicting the performance of different optimization algorithms for previously unseen problem instances is crucial for high-performing algorithm selection and configuration techniques. In the context of numerical optimization, supervised regression approaches built on top of exploratory landscape analysis are becoming very popular. From the point of view of Machine Learning (ML), however, the approaches are often rather naïve, using default regression or classification techniques without proper investigation of the suitability of the ML tools. With this work, we bring to the attention of our community the possibility to personalize regression models to specific types of optimization problems. Instead of aiming for a single model that works well across a whole set of possibly diverse problems, our personalized regression approach acknowledges that different models may suite different types of problems. Going one step further, we also investigate the impact of selecting not a single regression model per problem, but personalized ensembles. We test our approach on predicting the performance of numerical optimization heuristics on the BBOB benchmark collection.

The Special Sessions and Competitions on Real-Parameter Single Objective Optimization are benchmarking competitions held every year since 2013 that are used to evaluate the performance of new optimization algorithms.One flaw of these competitions is that algorithms are compared only to other algorithms submitted in the same year, not with algorithms submitted in previous years of the competition, so it can make comparison between all algorithms troublesome. Almost every year uses different benchmark functions, so the results between the years are not directly comparable. As a result, the winner of the most recent competition might not necessarily be significantly better than the winners of previous years.In this article, we directly compare winners of every competition held from 2013 to 2018 and present the results of this comparison. We use a benchmark set that consists of all test functions used by the competition throughout these years. We compare them on benchmark functions grouped by dimension (10, 30, 50, 100) and by year (2013, 2014, 2015, 2017). This allows us on one hand to see which algorithms perform best at specific dimensions, while grouping by year shows effects of parameter tuning on the end results.We present the results of these comparisons and find that later competition winners are not statistically better than algorithms from previous years in a general sense on every problem and dimensionality.

In this paper a novel statistical approach for comparing meta-heuristic stochastic optimization algorithms according to the distribution of the solutions in the search space is introduced, known as extended Deep Statistical Comparison. This approach is an extension of the recently proposed Deep Statistical Comparison approach used for comparing meta-heuristic stochastic optimization algorithms according to the solutions values. Its main contribution is that the algorithms are compared not only according to obtained solutions values, but also according to the distribution of the obtained solutions in the search space. The information it provides can additionally help to identify exploitation and exploration powers of the compared algorithms. This is important when dealing with a multimodal search space, where there are a lot of local optima with similar values. The benchmark results show that our proposed approach gives promising results and can be used for a statistical comparison of meta-heuristic stochastic optimization algorithms according to solutions values and their distribution in the search space.

Geographical information systems are ideal candidates for the application of parallel programming techniques, mainly because they usually handle large data sets. To help us deal with complex calculations over such data sets, we investigated the performance constraints of a classic master–worker parallel paradigm over a message-passing communication model. To this end, we present a new approach that employs an external database in order to improve the calculation–communication overlap, thus reducing the idle times for the worker processes. The presented approach is implemented as part of a parallel radio-coverage prediction tool for the Geographic Resources Analysis Support System (GRASS) environment. The prediction calculation employs digital elevation models and land-usage data in order to analyze the radio coverage of a geographical area. We provide an extended analysis of the experimental results, which are based on real data from an Long Term Evolution (LTE) network currently deployed in Slovenia. Based on the results of the experiments, which were performed on a computer cluster, the new approach exhibits better scalability than the traditional master–worker approach. We successfully tackled real-world-sized data sets, while greatly reducing the processing time and saturating the hardware utilization.

Continuous ant-colony optimization is an emerging field in numerical optimization, which tries to cope with the optimization challenges arising in modern real-world engineering and scientific domains. One of them is large-scale continuous optimization problem that becomes especially important for the development of recent emerging fields like bio-computing, data mining and production planing. Ant-colony optimization (ACO) is known for its efficiency in solving combinatorial optimization problems. However, its application to real-parameter optimizations appears more challenging, since the pheromone-laying method is not straightforward. In the recent year, there have been developed a several adaptations of the ACO algorithm for continuous optimization. Among them the Continuous Differential Ant-Stigmergy Algorithm (CDASA) arises as promising method for global continuous large-scale optimization. In this paper we address a systematic performance evaluation of CDASA on a predefined test suite and experimental procedure provided for the Competition on Real-Parameter Single Objective Optimization at CEC-2013.

In this paper a static network simulator is used to find downlink and uplink SHO areas. By introducing a penalty-based objective function and some hard constraints, we formally define the problem of balancing SHO areas in UMTS networks. The state-of-the-art mathematical model used and the penalty scores of the objective function are set according to the configuration and layout of a real mobile network, deployed in Slovenia by Telekom Slovenije, d.d.. The balancing problem is then tackled by three optimization algorithms, each of them belonging to a different category of metaheuristics. We report and analyze the optimization results, as well as the performance of each of the optimization algorithms used.

Many real-world problems are dynamic, requiring an optimization algorithm which is able to continuously track a changing optimum over time. In this paper, we present an ant-colony based algorithm for solving optimization problems with continuous variables, labeled Continuous Differential Ant-Stigmergy Algorithm (CDASA). The CDASA is applied to dynamic optimization problems without any modification to the algorithm. The performance of the CDASA is evaluated on the set of benchmark problems provided for the IEEE Competition on Evolutionary Computation for Dynamic Optimization Problems (ECDOP-Competition-2012).

In this paper, an optimization algorithm is proposed and its performance assessment for bound and equality/inequality constrained numerical optimization is presented. The proposed algorithm is called Continuous Differential Ant-Stigmergy Algorithm and is derived from the Differential Ant-Stigmergy Algorithm, which transforms a real-parameter optimization problem into a graph-search problem. The original algorithm is extended to use arbitrary real offsets to navigate through the search space. Experimental results are given for the Real World Optimization Problems proposed for the Special Session on Testing Evolutionary Algorithms on Real-world Numerical Optimization Problems at 2011 IEEE Congress on Evolutionary Computation.

Ant-colony optimization (ACO) is a popular swarm intelligence metaheuristic scheme that can be applied to almost any optimization problem. In this paper, we address a performance evaluation of an ACO-based algorithm for solving large-scale global optimization problems with continuous variables, labeled Differential Ant-Stigmergy Algorithm (DASA). The DASA transforms a real-parameter optimization problem into a graph-search problem. The parameters' differences assigned to the graph vertices are used to navigate through the search space. The performance of the DASA is evaluated on the set of benchmark problems provided for CEC'2010 Special Session and Competition on Large-Scale Global Optimization.

Many real-world problems are dynamic, requiring an optimization algorithm which is able to continuously track a changing optimum over time. In this paper, we present a stigmergy-based algorithm for solving optimization problems with continuous variables, labeled differential ant-stigmergy algorithm (DASA). The DASA is applied to dynamic optimization problems without any modification to the algorithm. The performance of the DASA is evaluated on the set of benchmark problems provided for CEC'2009 Special Session on Evolutionary Computation in Dynamic and Uncertain Environments.

This paper describes the so-called Differential Ant-Stigmergy Algorithm (DASA), which is an extension of the Ant-Colony Optimization for a continuous domain. An experimental evaluation of the DASA on a benchmark suite from CEC 2005 is presented. The DASA is compared with a number of evolutionary optimization algorithms, including the covariance matrix adaptation evolutionary strategy, the differential evolution, the real-coded memetic algorithm, and the continuous estimation of distribution algorithm. The DASA is also compared to some other ant methods for continuous optimization. The experimental results demonstrate the promising performance of the new approach. Besides this experimental work, the DASA was applied to a real-world problem, where the efficiency of the radial impeller of a vacuum cleaner was optimized. As a result the aerodynamic power was increased by twenty per cent.

The paper presents a new distributed metaheuristic algorithm in an optimal control problem related to the cooling process in the continuous casting of steel. The optimization task is to tune 18 coolant flows in the caster secondary cooling system to achieve the target surface temperatures along the slab. Sequential search algorithms are proved inefficient for this problem because they take too much time to compute an appropriate solution. For this reason a new distributed search algorithm based on stigmergy perceived in ant colony was developed. The algorithm was run on the grid that allows us to solve this optimization problem in much shorter time. As a matter of fact, the computation time can be decreased from half a day to a few hours without any decrease in the solution quality.