Objective functions with sharp ridge shapes are known to be difficult to optimize because they frequently lead to premature convergence near the ridges. To address this, local supremum transformations (LST) have been proposed as a method to mitigate sharpness by replacing the objective function value with the supremum over a local neighborhood. However, LST faces several challenges, including evaluation costs that increase linearly with the number of dimensions and failure when optimizing high-dimensional sharp functions. Therefore, as an efficient method for high-dimensional sharp functions, this paper proposes a hierarchical optimization method that utilizes two optimizers: an inner optimizer prone to premature convergence near the ridges, and a meta-optimizer that utilizes the result of the inner optimizer. The proposed method leverages the best solution found by the inner optimizer to update the meta-optimizer, which transforms problems with sharp shapes into a smooth objective function. Furthermore, by applying LST specifically in the evaluation of the best solution from the inner optimizer, the impact of sharp function shapes is mitigated. The experimental evaluation using (1+1)-CMA-ES as both inner and meta-optimizers demonstrates that the proposed method is effective for high-dimensional benchmark functions with sharp ridges compared to LST.

Mixed-integer extensions of evolution strategies (ES) that discretize selected coordinates of sampled continuous vectors often impose a lower bound on the standard deviation of integer variables to prevent premature convergence. While these methods show promising empirical results, this handling can slow the convergence of continuous variables, and its impact has lacked a clear theoretical account. In this paper, we provide a convergence analysis of evolution strategies for mixed-integer optimization, inspired by the drift analysis of the (1+1)-ES in the continuous domain. Specifically, we consider two (1+1)-ES variants for mixed-integer domains: (1+1)-LB-ES, which introduces a lower bound on the standard deviation for integer variables, and (1+1)-LUB-ES, which combines both lower and upper bounds to enhance the convergence of the continuous variables. Focusing on the optimization phase after the integer variables have been optimized, we rigorously analyze their convergence behavior on a benchmark function designed for mixed-integer domains. Our results show that (1+1)-LB-ES can suffer from premature convergence when the number of integer variables is large, while (1+1)-LUB-ES achieves linear convergence under suitable parameter settings. These findings provide theoretical insights into the impact of integer handling on convergence performance and guidance for the design of mixed-integer ES.

The augmented Lagrangian covariance matrix adaptation evolution strategy (AL-CMA-ES) is an efficient optimization method for constrained continuous black-box optimization problems, which integrates the Augmented Lagrangian method into CMA-ES. AL-CMA-ES incorporates penalty terms for the constraints into the objective function value and determines the ranking of solutions based on their evaluation value with penalties. AL-CMA-ES also adaptively tunes the penalty coefficients to balance the improvement of the objective and the constraint satisfaction. However, when multiple constraints are imposed, the coefficient adaptation is often delayed, which requires a fast movement of the search distribution after the coefficients are increased. In this paper, we introduce an element-wise effectiveness estimation mechanism of the update direction in AL-CMA-ES. This mechanism was originally proposed as a countermeasure for low effective dimensionality, a property where only a part of the design variables affects the evaluation value. This study verifies the effectiveness of the element-wise effectiveness estimation mechanism by accelerating the movement of the search distribution after delayed coefficient adaptation. Experimental results on the BBOB-constraint test suite demonstrated that the proposed method outperforms AL-CMA-ES with a large number of constraints.

Optimization problems in real-world applications across the medical and engineering domains often involve potential risks when evaluating candidate solutions. Safe optimization aims to perform optimization while suppressing unsafe solution evaluations in such situations. For continuous search spaces, there exist safe optimization methods based on evolutionary computation. However, the algorithm development of safe optimization methods for binary search spaces has not been adequately addressed. In this study, we incorporate additional mechanisms for safe optimization into a binary optimization method, the adaptive stochastic natural gradient method (ASNG) with a family of Bernoulli distributions. For safety functions that must be kept non-negative during optimization, the proposed method, safe ASNG, estimates the Lipschitz constants with respect to the Hamming distance by constructing surrogate models of safety functions based on discrete Walsh functions. Then, safe ASNG computes a safe region that consists of safe solutions around the previously evaluated safe solutions. By projecting newly generated solutions to their nearest neighbors within the safe region, safe ASNG suppresses unsafe solution evaluations. Experimental results on benchmark problems on binary domains confirm that, while the comparative methods fail to suppress unsafe solution evaluations, safe ASNG achieves efficient optimization while effectively suppressing unsafe solution evaluations.

This study focuses on mixed-variable black-box optimization (MV-BBO), addressing continuous, integer, and categorical variables. Many real-world MV-BBO problems involve dependencies among these different types of variables, requiring efficient methods to optimize them simultaneously. Recently, stochastic optimization methods leveraging the mechanism of the covariance matrix adaptation evolution strategy have shown promising results in mixed-integer or mixed-category optimization. However, such methods cannot handle the three types of variables simultaneously. In this study, we propose CatCMA with Margin (CatCMAwM), a stochastic optimization method for MV-BBO that jointly optimizes continuous, integer, and categorical variables. CatCMAwM is developed by incorporating a novel integer handling into CatCMA, a mixed-category black-box optimization method employing a joint distribution of multivariate Gaussian and categorical distributions. The proposed integer handling is carefully designed by reviewing existing integer handlings and following the design principles of CatCMA. Even when applied to mixed-integer problems, it stabilizes the marginal probability and improves the convergence performance of continuous variables. Numerical experiments show that CatCMAwM effectively handles the three types of variables, outperforming state-of-the-art Bayesian optimization methods and baselines that simply incorporate existing integer handlings into CatCMA.

This study focuses on the search space composed of disjoint sub-spaces, each containing common or distinct finite points on Euclidean space. This problem setting is called an optimization problem on sets of points (SoP), and acceptable solutions are constructed by selecting possible points in the subspaces. In optimization on SoP, it is essential to capture the positional relation between the points. Recently, CMA-ES-SoP was proposed as an optimization method on SoP by introducing additional mechanisms based on the Delaunay diagram to CMA-ES. However, there are two problems: the worst-case complexity of the Delaunay diagram is exponential in the number of dimensions, and the convergence of CMA-ES-SoP is relatively slow because of the non-elitist strategy. In this study, we propose an elitist evolutionary algorithm for the optimization on SoP. The proposed method, (1+1)-EA-SoP, adaptively switches two mutation methods; the neighboring-point mutation selects the mutated point from the neighbors on the graph, and the global mutation randomly selects one point. In addition, we develop a novel graph structure that can be constructed with polynomial complexity and possesses several desirable properties related to the Delaunay diagram. The experimental results show that (1+1)-EA-SoP with the proposed graph realizes an effective optimization on SoP.

High-dimensional optimization problems in real-world applications often possess the property called low effective dimensionality (LED), where only a small part of directions in search space affect the evaluation value, and others are redundant. On problems with LED, because the redundant directions deteriorate the prediction performance of the surrogate model, the performance of several surrogate-assisted evolutionary algorithms is worsened. This paper focuses on the doubly trained surrogate CMA-ES (DTS-CMA-ES) that employs Gaussian process regression as a surrogate model and proposes DTS-CMA-ES-LED by incorporating several countermeasures for LED to DTS-CMA-ES. The proposed method considers directions along the eigenvectors of the covariance matrix and evaluates the effectiveness of each direction using the estimated element-wise signal-to-noise ratio of the update directions. Then, the proposed method reconstructs the kernel function with the computed effectiveness to reduce the effect of redundant directions. We also introduce the hyperparameter adaptation mechanism and refinement of the step-size adaptation as countermeasures for LED. The experimental results show that DTS-CMA-ES-LED effectively optimized the benchmark functions with LED.

Bulldozer automation is becoming increasingly important to address the shortage of skilled operators and to improve safety in construction and mining operations. Self-Localization is a key requirement for achieving such automation. However, ensuring reliable self-localization in environments where Global Navigation Satellite System (GNSS) signals are frequently unavailable remains a major challenge. While machine learning-based self-localization methods have gained significant attention, their inherent prediction uncertainties must be accounted for in practical applications. This paper presents a novel self-localization system that relies solely on internal sensors and accounts for both epistemic and aleatoric uncertainties in machine learning predictions. The proposed method first employs a machine learning model to estimate local velocity and its uncertainty, using Deep Ensembles and Gaussian Maximum Likelihood Training. These estimates are then integrated by an Extended Kalman Filter to determine global position. We evaluated our approach using real-world data from an actual bulldozer across diverse operating conditions, such as slope traversal and slalom maneuvers. Experimental results show that our uncertainty-aware method produces more plausible and reliable position estimates than baseline methods that do not account for uncertainty. The proposed system offers a cost-effective solution for enhancing construction equipment autonomy in GNSS-denied environments.

The majority of theoretical analyses of evolutionary algorithms in the discrete domain focus on binary optimization algorithms, even though black-box optimization on the categorical domain has many practical applications. In this paper, we consider a probabilistic model-based algorithm using the family of categorical distributions as its underlying distribution and set the sample size as two. We term this specific algorithm the categorical compact genetic algorithm (ccGA). The ccGA can be considered as an extension of the compact genetic algorithm (cGA), which is an efficient binary optimization algorithm. We theoretically analyze the dependency of the number of possible categories K, the number of dimensions D, and the learning rate η on the runtime. We investigate the tail bound of the runtime on two typical linear functions on the categorical domain: categorical OneMax (COM) and KVal. We derive that the runtimes on COM and KVal are O(Dln(DK)/η) and Θ(DlnK/η) with high probability, respectively. Our analysis is a generalization for that of the cGA on the binary domain.

Black-box optimization problems often require simultaneously optimizing different types of variables, such as continuous, integer, and categorical variables. Unlike integer variables, categorical variables do not necessarily have a meaningful order, and the discretization approach of continuous variables does not work well. Although several Bayesian optimization methods can deal with mixed-category black-box optimization (MC-BBO), they suffer from a lack of scalability to high-dimensional problems and internal computational cost. This paper proposes CatCMA, a stochastic optimization method for MC-BBO problems, which employs the joint probability distribution of multivariate Gaussian and categorical distributions as the search distribution. CatCMA updates the parameters of the joint probability distribution in the natural gradient direction. CatCMA also incorporates the acceleration techniques used in the covariance matrix adaptation evolution strategy (CMA-ES) and the stochastic natural gradient method, such as step-size adaptation and learning rate adaptation. In addition, we restrict the ranges of the categorical distribution parameters by margin to prevent premature convergence and analytically derive a promising margin setting. Numerical experiments show that the performance of CatCMA is superior and more robust to problem dimensions compared to state-of-the-art Bayesian optimization algorithms.

In several real-world applications in medical and control engineering, there are unsafe solutions whose evaluations involve inherent risk. This optimization setting is known as safe optimization and formulated as a specialized type of constrained optimization problem with constraints for safety functions. Safe optimization requires performing efficient optimization without evaluating unsafe solutions. A few studies have proposed the optimization methods for safe optimization based on Bayesian optimization and the evolutionary algorithm. However, Bayesian optimization-based methods often struggle to achieve superior solutions, and the evolutionary algorithm-based method fails to effectively reduce unsafe evaluations. This study focuses on CMA-ES as an efficient evolutionary algorithm and proposes an optimization method termed safe CMA-ES. The safe CMA-ES is designed to achieve both safety and efficiency in safe optimization. The safe CMA-ES estimates the Lipschitz constants of safety functions transformed with the distribution parameters using the maximum norm of the gradient in Gaussian process regression. Subsequently, the safe CMA-ES projects the samples to the nearest point in the safe region constructed with the estimated Lipschitz constants. The numerical simulation using the benchmark functions shows that the safe CMA-ES successfully performs optimization, suppressing the unsafe evaluations, while the existing methods struggle to significantly reduce the unsafe evaluations.

The covariance matrix adaptation evolution strategy (CMA-ES) is a powerful optimization method for continuous black-box optimization problems. Several noise-handling methods have been proposed to bring out the optimization performance of the CMA-ES on noisy objective functions. The adaptations of the population size and the learning rate are two major approaches that perform well under additive Gaussian noise. The reevaluation technique is another technique that evaluates each solution multiple times. In this paper, we discuss the difference between those methods from the perspective of stochastic relaxation that considers the maximization of the expected utility function. We derive that the set of maximizers of the noise-independent utility, which is used in the reevaluation technique, certainly contains the optimal solution, while the noise-dependent utility, which is used in the population size and leaning rate adaptations, does not satisfy it under multiplicative noise. Based on the discussion, we develop the reevaluation adaptation CMA-ES (RA-CMA-ES), which computes two update directions using half of the evaluations and adapts the number of reevaluations based on the estimated correlation of those two update directions. The numerical simulation shows that the RA-CMA-ES outperforms the comparative method under multiplicative noise, maintaining competitive performance under additive noise.

Motivated by the high prediction performance of convolutional neural networks (CNNs), several works have applied them to tabular datasets. As CNNs are built to accept images, several transformations of tabular data have been proposed to obtain images. However, existing methods transform the tabular data into images prior to CNN training, which fails to take the prediction error into account. Additionally, they employ all features from the tables, including unimportant ones, to produce the images. Moreover, the created images might not become human-interpretable because they do not consider the interpretability of images as a metric. To overcome these problems, we propose a hard attention-based converter combined with a convolutional neural network (HACNet), consisting of an attention-based table-to-image converter and a CNN-based predictor. HACNet trains its components simultaneously by minimizing CNN prediction loss and mean squared error (MSE) between created and template images. Minimizing this MSE loss allows us to visually distinguish the created images with different labels. The attention-based converter selects exactly one feature for each pixel in the image via its hard attention mechanism with Gumbel-Softmax, enabling feature selection. We experimentally show that HACNet produces human-interpretable images, reduces used features, and achieves prediction performances comparative with existing methods on several benchmark datasets.

The covariance matrix adaptation evolution strategy (CMA-ES) is an efficient continuous black-box optimization method. The CMA-ES possesses many attractive features, including invariance properties and a well-tuned default hyperparameter setting. Moreover, several components to specialize the CMA-ES have been proposed, such as noise handling and constraint handling. To utilize these advantages in mixed-integer optimization problems, the CMA-ES with margin has been proposed. The CMA-ES with margin prevents the premature convergence of discrete variables by the margin correction, in which the distribution parameters are modified to leave the generation probability for changing the discrete variable. The margin correction has been applied to (μ/μw,Λ)-CMA-ES, while this paper introduces the margin correction into (1+1)-CMA-ES, an elitist version of CMA-ES. The (1+1)-CMA-ES is often advantageous for unimodal functions and can be computationally less expensive. To tackle the performance deterioration on mixed-integer optimization, we use the discretized elitist solution as the mean of the sampling distribution and modify the margin correction not to move the elitist solution. The numerical simulation using benchmark functions on mixed-integer, integer, and binary domains shows that (1+1)-CMA-ES with margin outperforms the CMA-ES with margin and is better than or comparable with several specialized methods to a particular search domain.

Understanding the relationship between material characteristics and microscopic structure is important for the development of solder materials. To clarify this relationship, machine learning approaches are often used to predict material characteristics from microstructural images. Although a trained machine learning model can predict the material characteristics for a given microstructural image, it cannot directly create microstructural images of solder materials with desirable characteristics. Therefore, it is difficult to use machine learning to develop new solder materials. This paper presents a method for generating electron probe micro-analyzer (EPMA) images of a solder with desirable characteristics using deep learning. Our method uses a generative adversarial network (GAN) to generate images, and a convolutional neural network (CNN)-based evaluator to predict their characteristics. A common difficulty in applying machine learning to material science is the lack of training data, which often results in predictions with low accuracy. To address the small dataset problem, we trained the ranking prediction model of the characteristics instead of the regression model. Moreover, we employed transfer learning, in which a CNN model trained on texture datasets was used as the initial model. The experimental results show that the GAN successfully generated EPMA images that were similar to the actual images. The use of the ranking prediction model and transfer learning improved the performance of the CNN-based characteristic evaluator. We then selected promising generated EPMA images using the CNN-based characteristic evaluator and found that the characteristics of the selected EPMA images were consistent with expert experience.

We theoretically analyze the information geometric optimization (IGO), which is a unified framework of stochastic search algorithms for black-box optimization. The IGO framework has two parameters: 1) the learning rate and 2) the sample size, and they influence the behavior of the algorithm. We investigate the strategy parameters of the IGO with the family of isotropic Gaussian distributions on a general convex quadratic function. Compared to the previous theoretical works, where an infinite sample size is assumed and the deterministic algorithm dynamics is studied, we investigate the expected improvement of the algorithm with a finite sample size. The analysis finds that the relative decrease rates of the distance from the distribution mean to the landscape optimum and the distribution standard deviation must be the same, which we observe in practice, while the analysis based on an infinite sample size failed to obtain. We derive these rates explicitly as a function of the eigenvalues of the Hessian of the objective function and the strategy parameters. We also derive the stable value of the ratio of the square distance to the optimum over the distribution variance, as well as the conditions that the stable value exists. These theoretical values coincide with our numerical simulations.

High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-applicable NAS, we develop a generic optimization framework for NAS. We turn a coupled optimization of connection weights and neural architecture into a differentiable optimization by means of stochastic relaxation. It accepts arbitrary search space (widely-applicable) and enables to employ a gradient-based simultaneous optimization of weights and architecture (fast). We propose a stochastic natural gradient method with an adaptive step-size mechanism built upon our theoretical investigation (robust). Despite its simplicity and no problem-dependent parameter tuning, our method exhibited near state-of-the-art performances with low computational budgets both on image classification and inpainting tasks.

In this article, we theoretically investigate the convergence properties of the information geometric optimization (IGO) algorithm given the family of isotropic Gaussian distributions on the sphere function. Differently from previous studies, where the exact natural gradient is taken, i.e., the infinite samples are assumed, we consider the case that the natural gradient is estimated from finite samples. We derive the rates of the expected decrease of the squared distance to the optimum and the variance parameter as functions of the learning rates, dimension, and sample size. From the rates of decrease deduces that the rates of decreases of the squared distance to the optimum and the variance parameter must agree for geometric convergence of the algorithm. In other words, the ratio between the squared distance to the optimum and the variance must be stable, which is observed empirically but is not derived in the previous theoretical studies. We further derive the condition on the learning rates that the rates of decreases agree and derive the stable value of the ratio. We confirm in simulation that the derived rates of decreases and the stable value of the ratio well approximate the behavior of the IGO algorithm.