Machine learning (ML) systems are utilized in critical sectors such as healthcare, law enforcement, and finance, but often rely on historical data that contains demographic biases, leading to decisions that perpetuate or intensify existing inequalities. Causal and counterfactual fairness provide a transparent, human-in-the-loop framework to mitigate algorithmic discrimination, aligning closely with legal doctrines of direct and indirect discrimination. However, current causal fairness frameworks hold a key limitation in that they assume prior knowledge of the correct causal model, restricting their applicability in complex fairness scenarios where causal models are unknown or difficult to identify. To bridge this gap, we propose FairPFN, a tabular foundation model pre-trained on synthetic causal fairness data to identify and mitigate the causal effects of protected attributes in its predictions. FairPFN’s key contribution is that it requires no knowledge of the causal model and demonstrates strong performance across a diverse set of hand-crafted and real-world causal scenarios relative to robust baseline methods. FairPFN paves the way for a promising direction for future research, making causal fairness more accessible to a wider variety of complex fairness problems.

Fairness-aware Machine Learning (FairML) applications are often characterized by complex social objectives and legal requirements, frequently involving multiple, potentially conflicting notions of fairness. Despite the well-known Impossibility Theorem of Fairness and extensive theoretical research on the statistical and socio-technical trade-offs between fairness metrics, many FairML tools still optimize or constrain for a single fairness objective. However, this one-sided optimization can inadvertently lead to violations of other relevant notions of fairness. In this socio-technical and empirical study, we frame fairness as a Many-Objective (MaO) problem by treating fairness metrics as conflicting objectives in a multi-objective (MO) sense. We introduce ManyFairHPO, a human-in-the-loop, fairness-aware model selection framework that enables practitioners to effectively navigate complex and nuanced fairness objective landscapes. ManyFairHPO aids in the identification, evaluation, and balancing of fairness metric conflicts and their related social consequences, leading to more informed and socially responsible model-selection decisions. Through a comprehensive empirical evaluation and a case study on the Law School Admissions problem, we demonstrate the effectiveness of ManyFairHPO in balancing multiple fairness objectives, mitigating risks such as self-fulfilling prophecies, and providing interpretable insights to guide stakeholders in making fairness-aware modeling decisions.

The field of automated machine learning (AutoML) introduces techniques that automate parts of the development of machine learning (ML) systems, accelerating the process and reducing barriers for novices. However, decisions derived from ML models can reproduce, amplify, or even introduce unfairness in our societies, causing harm to (groups of) individuals. In response, researchers have started to propose AutoML systems that jointly optimize fairness and predictive performance to mitigate fairness-related harm. However, fairness is a complex and inherently interdisciplinary subject, and solely posing it as an optimization problem can have adverse side effects. With this work, we aim to raise awareness among developers of AutoML systems about such limitations of fairness-aware AutoML, while also calling attention to the potential of AutoML as a tool for fairness research. We present a comprehensive overview of different ways in which fairness-related harm can arise and the ensuing implications for the design of fairness-aware AutoML. We conclude that while fairness cannot be automated, fairness-aware AutoML can play an important role in the toolbox of ML practitioners. We highlight several open technical challenges for future work in this direction. Additionally, we advocate for the creation of more user-centered assistive systems designed to tackle challenges encountered in fairness work. This article appears in the AI & Society track.

Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL). Practitioners are often faced with the trade-off between multiple criteria, such as accuracy and latency. Given the high computational needs of DL and the growing demand for efficient HPO, the acceleration of multi-objective (MO) optimization becomes ever more important. Despite the significant body of work on meta-learning for HPO, existing methods are inapplicable to MO tree-structured Parzen estimator (MO-TPE), a simple yet powerful MO-HPO algorithm. In this paper, we extend TPE’s acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. We also theoretically analyze and address the limitations of our task similarity. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on “Multiobjective Hyperparameter Optimization for Transformers”. See https://arxiv.org/abs/2212.06751 for the latest version with Appendix.

The performance of an algorithm often critically depends on its parameter configuration. While a variety of automated algorithm configuration methods have been proposed to relieve users from the tedious and error-prone task of manually tuning parameters, there is still a lot of untapped potential as the learned configuration is static, i.e., parameter settings remain fixed throughout the run. However, it has been shown that some algorithm parameters are best adjusted dynamically during execution. Thus far, this is most commonly achieved through hand-crafted heuristics. A promising recent alternative is to automatically learn such dynamic parameter adaptation policies from data. In this article, we give the first comprehensive account of this new field of automated dynamic algorithm configuration (DAC), present a series of recent advances, and provide a solid foundation for future research in this field. Specifically, we (i) situate DAC in the broader historical context of AI research; (ii) formalize DAC as a computational problem; (iii) identify the methods used in prior art to tackle this problem; and (iv) conduct empirical case studies for using DAC in evolutionary optimization, AI planning, and machine learning.

Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever. Here, we combine the advantages of the popular bandit-based HPO method Hyperband (HB) and the evolutionary search approach of Differential Evolution (DE) to yield a new HPO method which we call DEHB. Comprehensive results on a very broad range of HPO problems, as well as a wide range of tabular benchmarks from neural architecture search, demonstrate that DEHB achieves strong performance far more robustly than all previous HPO methods we are aware of, especially for high-dimensional problems with discrete input dimensions. For example, DEHB is up to 1000x faster than random search. It is also efficient in computational time, conceptually simple and easy to implement, positioning it well to become a new default HPO method.

This paper introduces an enhancement of the recent developed Gaining Sharing Knowledge based algorithm, dubbed as GSK. This algorithm is an excellent example of a contemporary nature-based algorithm which is inspired from the human life behavior of gaining and sharing knowledge to solve the optimization task. GSK algorithm simulates the natural phenomena of human gaining and sharing knowledge using two main phases: junior and senior. A set of initial solutions are generated at the beginning of the search which are consideredjuniors. Later, the individuals are moving to senior stage by interacting with the environment and cooperating with other solutions during the search. The key idea in this work is to extend and improve the original GSK algorithm by proposing adaptive settings to the two important control parameters: knowledge factor and knowledge ratio. These two parameters are responsible to control junior and senior gaining and sharing phases between the solutions during the optimization loop. The algorithm is named AGSK and tested on the recent benchmark suite on bound constrained numerical optimization which consists of different challenging optimization problems with different dimensions. This benchmark is presented in IEEE-CEC2020 competition. When compared with other state-of-the-art algorithms including original GSK, AGSK shows superior performance.

Recently, Evolution Strategies (ES) have been successfully applied to solve problems commonly addressed by reinforcement learning (RL). Due to the simplicity of ES approaches, their runtime is often dominated by the RL-task at hand (e.g., playing a game). In this work, we introduce Progressive Episode Lengths (PEL) as a new technique and incorporate it with ES. The main objective is to allow the agent to play short and easy tasks with limited lengths, and then use the gained knowledge to further solve long and hard tasks with progressive lengths. Hence allowing the agent to perform many function evaluations and find a good solution for short time horizons before adapting the strategy to tackle larger time horizons. We evaluated PEL on a subset of Atari games from OpenAI Gym, showing that it can substantially improve the optimization speed, stability and final score of canonical ES. Specifically, we show average improvements of 80% (32%) after 2 hours (10 hours) compared to canonical ES.

In smart grid, energy resource management is highly complex large-scale optimization problem where the aim is to maximize the incomes while minimizing operational costs. Due to presence of mixed-integer variables and non-linear constraints, recently the use of evolutionary algorithms as a tool to find optimal and near-optimal solutions is becoming popular. The energy resource management problem further gets complicated if the uncertainty associated with the renewable generation, load forecast errors, electric vehicles scheduling and market prices are considered. Therefore, in a real-world scenario, it is essential to address the issues brought by the variability of demand, renewable energy, electric vehicles, and market price variations while maximizing the incomes and minimizing the total operation costs. In this paper, we analyze the performance of Differential Evolution with a stochastic selection on a large-scale energy resource management problem with uncertainty designed for competition at CEC 2018. The system comprises of a 25-bus microgrid representing a residential area with high penetration of Distributed Energy Resources (DER), Electric Vehicles (EVs), Demand Response (DR) programs etc.

Many Differential Evolution algorithms are introduced in the literature to solve optimization problems with diverse set of characteristics. In this paper, we propose an extension of the previously published paper LSHADE-EpSin that was ranked as the joint winner in the real-parameter single objective optimization competition, CEC 2016. The contribution of this work constitutes two major modifications that have been added to enhance the performance: ensemble of sinusoidal approaches based on performance adaptation and covariance matrix learning for the crossover operator. Two sinusoidal waves have been used to adapt the scaling factor: non-adaptive sinusoidal decreasing adjustment and an adaptive sinusoidal increasing adjustment. Instead of choosing one of the sinusoidal waves randomly, a performance adaptation scheme based on earlier success is used in this work. Moreover, covariance matrix learning with Euclidean neighborhood is used for the crossover operator to establish a suitable coordinate system, and to enhance the capability of LSHADE-EpSin to tackle problems with high correlation between the variables. The proposed algorithm, namely LSHADE-cnEpSin, is tested on the IEEE CEC2017 problems used in the Special Session and Competitions on Single Objective Bound Constrained Real-Parameter Single Objective Optimization. The results statistically affirm the efficiency of the proposed approach to obtain better results compared to other state-of-the-art algorithms.

Solving real-world optimization problems is considered a challenging task. This is due to the variability of the characteristics in objective functions, the presence of enormous number of local optima within the search space and highly nonlinear constraints with large number of variables. The advances on this type of problems are of capital importance for many researchers to develop new efficient evolutionary algorithms to tackle such problems in an efficient manner with better solutions. For this reason, this work proposes a new crossover technique based on covariance learning with Euclidean neighborhood which has been incorporated in the basic L-SHADE algorithm. The goal of this new technique is to help L-SHADE establish a suitable coordinate system for the crossover operator. This helps enhance L-SHADE capability to solve real world problems with difficult characteristics and nonlinear constraints. The proposed algorithm, namely L-covnSHADE, is tested on one of the challenging benchmarks which is the IEEE CEC'11 on real-world numerical optimization problems. This set consists of 22 real-world problems with diverse stimulating characteristics and a dimensionality ranging from 1 to 240 dimensions. The results statistically affirm the efficiency of the proposed approach to obtain better results compared to the L-SHADE algorithm and other state-of-the-art algorithms including the winner of the CEC2011 competition.

Multilevel inverters are mainly used for DC to AC power conversion and these inverters can be classified into types current source inverter (CSI) and voltage source inverter (VSI). Voltage source inverters are more common in power industry to convert lower levels of DC voltages into higher levels of AC voltages. In the process of conversion widely implemented pulse width modulated (PWM) switching technique of DC sources introduces harmonics in inverter output voltage. Total harmonic distortion (THD) is a measure of harmonic pollution in the power system and it is observed that variations in both DC voltages and switching angles of inverter affect the THD of inverter output voltage. Cascaded multilevel symmetric inverters ideally have DC sources all equal and constant. This paper considers inverters where DC sources can be unequal, a justifiable and realistic supposition. Optimal values of DC voltages and switching angles, which minimize THD level, are found using evolutionary algorithm. An advanced form of Differential Evolution (DE), called LSHADE-EpSin, is applied for the optimization problem. SHADE is a success history based parameter adaptation technique of DE. LSHADE improves the performance of SHADE with linearly reducing the population size in successive generations. LSHADE-EpSin introduces an additional adaptation technique for control parameters of the evolutionary algorithm. The algorithm has successfully been implemented for higher levels of inverters considered in the scope of our research study.

Developing efficient evolutionary algorithms attracts many researchers due to the existence of optimization problems in numerous real-world applications. A new differential evolution algorithm, sTDE-dR, is proposed to improve the search quality, avoid premature convergence, and stagnation. The population is clustered in multiple tribes and utilizes an ensemble of different mutation and crossover strategies. In this algorithm, a competitive success-based scheme is introduced to determine the life cycle of each tribe and its participation ratio for the next generation. In each tribe, a different adaptive scheme is used to control the scaling factor and crossover rate. The mean success of each subgroup is used to calculate the ratio of its participation for the next generation. This guarantees that successful tribes with the best adaptive schemes are only the ones that guide the search toward the optimal solution. The population size is dynamically reduced using a dynamic reduction method. Comprehensive comparison of the proposed heuristic over a challenging set of benchmarks from the CEC2014 real parameter single objective competition against several state-of-the-art algorithms is performed. The results affirm robustness of the proposed approach compared to other state-of-the-art algorithms.

Evolutionary algorithms are successfully developed to handle the challenges in solving optimization problems with complex landscapes. Differential evolution proves its efficiency as a powerful evolutionary algorithm to solve complex optimization problems with diverse characteristics. In this paper, we aim at designing an enhanced evolutionary algorithm that embeds Differential Evolution in Stochastic Fractal Search. Stochastic Fractal Search is developed recently as a powerful metaheuristic algorithm that imitates the natural phenomenon of growth and uses the diffusion process based on random fractals. In this paper, we introduce a new adjustment to the Diffusion Process of Stochastic Fractal Search. The proposed algorithm namely, SFS-DPDE-GW, uses Differential Evolution in the Diffusion Process along with the Gaussian Walks to enhance the search. To validate the performance of our algorithm, a challenging test suite of 30 benchmark functions from the IEEE CEC2014 real parameter single objective competition is used. The proposed combination clearly enhances the performance of Stochastic Fractal Search and increases the efficiency of the update process which was incorporated after Diffusion process. Comparative studies show that the new algorithm has a superior performance compared to the original Stochastic Fractal Search and other recent state-of-the-art algorithms.

An effective and efficient self-adaptation framework is proposed to improve the performance of the L-SHADE algorithm by providing successful alternative adaptation for the selection of control parameters. The proposed algorithm, namely LSHADE-EpSin, uses a new ensemble sinusoidal approach to automatically adapt the values of the scaling factor of the Differential Evolution algorithm. This ensemble approach consists of a mixture of two sinusoidal formulas: A non-Adaptive Sinusoidal Decreasing Adjustment and an adaptive History-based Sinusoidal Increasing Adjustment. The objective of this sinusoidal ensemble approach is to find an effective balance between the exploitation of the already found best solutions, and the exploration of non-visited regions. A local search method based on Gaussian Walks is used at later generations to increase the exploitation ability of LSHADE-EpSin. The proposed algorithm is tested on the IEEE CEC2014 problems used in the Special Session and Competitions on Real-Parameter Single Objective Optimization of the IEEE CEC2016. The results statistically affirm the efficiency and robustness of the proposed approach to obtain better results compared to L-SHADE algorithm and other state-of-the-art algorithms.

This paper introduces a Cluster-based Differential Evolution Algorithm with Heterogeneous Influence for solving complex optimization problems. The idea behind this combination is to classify the Differential Evolution population into a number of clusters using k-means clustering method and to apply different mutation strategies for the clusters. The number of clusters is changed dynamically in each generation. The proposed algorithm uses three mutation strategies: DE/best-group/ 1/exp, DE/rand1/exp and DE/rand/1/bin. The DE/best-group/ 1/exp is an improved mutation strategy that randomly selects a portion of the population and then chooses the best individual in the group to guide the evolution. The k-means clustering algorithm is used periodically to fine-tune solutions that are generated from DE/best-group/1/exp by producing new clusters. This helps in balancing the exploration and exploitation capabilities by using different mutation strategies for these clusters to enhance diversity. The performance of the proposed approach is tested on 25 complex benchmark functions on single objective real-parameter numerical optimization. Results show that the proposed algorithm exhibits competitive performance when compared to other state-of-the-art algorithms.

Developing efficient evolutionary algorithms for solving learning-based real-parameter single objective optimization is a very challenging and essential task in many real applications. This task involves finding the best optimal solution with least computational cost, avoiding premature convergence. This paper proposes a new efficient Differential Evolution algorithm with success-based parameter adaptation with resizing population space. We introduce a new technique to adapt the control parameters which uses a memory-based structure of previous successful settings. Moreover, the population size is adapted linearly to find the most suitable size which helps to guide the search in each optimization loop. The proposed algorithm is tested on the benchmarks of the CEC2015 real parameter single objective competition. The results affirm the efficiency and robustness of our approach to reach good results.

Many meta-heuristics methods are applied to guide the exploration and exploitation of the search space for large scale optimization problems. These problems have attracted much attention from researchers who proposed developed a variety of techniques for locating the optimal solutions. Cultural Algorithm has been recently adopted to solve global numerical optimization problems. In this paper, a modified version of Cultural Algorithm (CA) that uses four knowledge sources in order to incorporate the information obtained from the objective function as well as constraint violation into knowledge structure in the belief space is proposed. The archived knowledge in the proposed approach will be used to enhance the way the belief space influences future generations of problem solvers. The first step is to use the four knowledge sources to guide the direction of the search to more promising solutions. The search is balanced between exploration and exploitation by dynamically adjusting the number of evaluations available for each type of knowledge source based on whether is primarily exploratory or exploitative. The second step selects one local search method to find the nearest solutions to those proposed by the knowledge sources. The proposed work is employed to solve seven global optimization problems in 50 and 100 dimensions, and an engineering application problem. Simulation results show how the approach speeds up the convergence process with very competitive results on such complex benchmarks when compared to other state-of-the-art algorithms.

Many evolutionary computational models have been introduced for solving engineering optimization problems that usually intend to find the global optimum solution. These methods, however, expose high computational effort and lack the diversity of the population and hence remain trapped in a local optimum. In this paper, we propose new hybrid optimization model, where a version of niche Cultural Algorithm is integrated with Tabu Search to guide the fittest individuals to new promising areas, aiming to escape local optima. The proposed approach significantly improves the performance of Cultural Algorithm by maintaining a high diversity among the population of problem solvers. This helps avoid premature and enhances located solutions. The technique is tested using a set of real-parameter optimization benchmark problems. The results in all cases indicate that the proposed method is capable of obtaining the optimal solutions with small number of function evaluations.

In this paper we propose an optimization algorithm for global optimization problems. The proposed algorithm is named (CA-ImLS) and is based on Cultural Algorithms and an improved local search approach for optimization over large-scale continuous spaces. In this paper, Cultural Algorithm and an improved sub-regional local search method are hybridized to form CA-ImLS. The original Cultural Algorithm is extended to have five parallel local searches that are rooted to its knowledge sources in the belief space component. This directs the search in multi-directions and improves the capability of its problem solvers in obtaining better-quality solutions. The distribution of new search agents is based on the success of the knowledge sources in which each knowledge source has its own local search for generating new agents with better fitness values and enhanced diversity to avoid stagnation. Experimental results are given for a set of benchmark optimization functions. Results indicate an average improvement of 2%-83% over the basic Cultural Algorithm framework.