Objective functions with sharp ridge shapes are known to be difficult to optimize because they frequently lead to premature convergence near the ridges. To address this, local supremum transformations (LST) have been proposed as a method to mitigate sharpness by replacing the objective function value with the supremum over a local neighborhood. However, LST faces several challenges, including evaluation costs that increase linearly with the number of dimensions and failure when optimizing high-dimensional sharp functions. Therefore, as an efficient method for high-dimensional sharp functions, this paper proposes a hierarchical optimization method that utilizes two optimizers: an inner optimizer prone to premature convergence near the ridges, and a meta-optimizer that utilizes the result of the inner optimizer. The proposed method leverages the best solution found by the inner optimizer to update the meta-optimizer, which transforms problems with sharp shapes into a smooth objective function. Furthermore, by applying LST specifically in the evaluation of the best solution from the inner optimizer, the impact of sharp function shapes is mitigated. The experimental evaluation using (1+1)-CMA-ES as both inner and meta-optimizers demonstrates that the proposed method is effective for high-dimensional benchmark functions with sharp ridges compared to LST.

Mixed-integer extensions of evolution strategies (ES) that discretize selected coordinates of sampled continuous vectors often impose a lower bound on the standard deviation of integer variables to prevent premature convergence. While these methods show promising empirical results, this handling can slow the convergence of continuous variables, and its impact has lacked a clear theoretical account. In this paper, we provide a convergence analysis of evolution strategies for mixed-integer optimization, inspired by the drift analysis of the (1+1)-ES in the continuous domain. Specifically, we consider two (1+1)-ES variants for mixed-integer domains: (1+1)-LB-ES, which introduces a lower bound on the standard deviation for integer variables, and (1+1)-LUB-ES, which combines both lower and upper bounds to enhance the convergence of the continuous variables. Focusing on the optimization phase after the integer variables have been optimized, we rigorously analyze their convergence behavior on a benchmark function designed for mixed-integer domains. Our results show that (1+1)-LB-ES can suffer from premature convergence when the number of integer variables is large, while (1+1)-LUB-ES achieves linear convergence under suitable parameter settings. These findings provide theoretical insights into the impact of integer handling on convergence performance and guidance for the design of mixed-integer ES.

The augmented Lagrangian covariance matrix adaptation evolution strategy (AL-CMA-ES) is an efficient optimization method for constrained continuous black-box optimization problems, which integrates the Augmented Lagrangian method into CMA-ES. AL-CMA-ES incorporates penalty terms for the constraints into the objective function value and determines the ranking of solutions based on their evaluation value with penalties. AL-CMA-ES also adaptively tunes the penalty coefficients to balance the improvement of the objective and the constraint satisfaction. However, when multiple constraints are imposed, the coefficient adaptation is often delayed, which requires a fast movement of the search distribution after the coefficients are increased. In this paper, we introduce an element-wise effectiveness estimation mechanism of the update direction in AL-CMA-ES. This mechanism was originally proposed as a countermeasure for low effective dimensionality, a property where only a part of the design variables affects the evaluation value. This study verifies the effectiveness of the element-wise effectiveness estimation mechanism by accelerating the movement of the search distribution after delayed coefficient adaptation. Experimental results on the BBOB-constraint test suite demonstrated that the proposed method outperforms AL-CMA-ES with a large number of constraints.

Optimization problems in real-world applications across the medical and engineering domains often involve potential risks when evaluating candidate solutions. Safe optimization aims to perform optimization while suppressing unsafe solution evaluations in such situations. For continuous search spaces, there exist safe optimization methods based on evolutionary computation. However, the algorithm development of safe optimization methods for binary search spaces has not been adequately addressed. In this study, we incorporate additional mechanisms for safe optimization into a binary optimization method, the adaptive stochastic natural gradient method (ASNG) with a family of Bernoulli distributions. For safety functions that must be kept non-negative during optimization, the proposed method, safe ASNG, estimates the Lipschitz constants with respect to the Hamming distance by constructing surrogate models of safety functions based on discrete Walsh functions. Then, safe ASNG computes a safe region that consists of safe solutions around the previously evaluated safe solutions. By projecting newly generated solutions to their nearest neighbors within the safe region, safe ASNG suppresses unsafe solution evaluations. Experimental results on benchmark problems on binary domains confirm that, while the comparative methods fail to suppress unsafe solution evaluations, safe ASNG achieves efficient optimization while effectively suppressing unsafe solution evaluations.

This study focuses on mixed-variable black-box optimization (MV-BBO), addressing continuous, integer, and categorical variables. Many real-world MV-BBO problems involve dependencies among these different types of variables, requiring efficient methods to optimize them simultaneously. Recently, stochastic optimization methods leveraging the mechanism of the covariance matrix adaptation evolution strategy have shown promising results in mixed-integer or mixed-category optimization. However, such methods cannot handle the three types of variables simultaneously. In this study, we propose CatCMA with Margin (CatCMAwM), a stochastic optimization method for MV-BBO that jointly optimizes continuous, integer, and categorical variables. CatCMAwM is developed by incorporating a novel integer handling into CatCMA, a mixed-category black-box optimization method employing a joint distribution of multivariate Gaussian and categorical distributions. The proposed integer handling is carefully designed by reviewing existing integer handlings and following the design principles of CatCMA. Even when applied to mixed-integer problems, it stabilizes the marginal probability and improves the convergence performance of continuous variables. Numerical experiments show that CatCMAwM effectively handles the three types of variables, outperforming state-of-the-art Bayesian optimization methods and baselines that simply incorporate existing integer handlings into CatCMA.

This study focuses on the search space composed of disjoint sub-spaces, each containing common or distinct finite points on Euclidean space. This problem setting is called an optimization problem on sets of points (SoP), and acceptable solutions are constructed by selecting possible points in the subspaces. In optimization on SoP, it is essential to capture the positional relation between the points. Recently, CMA-ES-SoP was proposed as an optimization method on SoP by introducing additional mechanisms based on the Delaunay diagram to CMA-ES. However, there are two problems: the worst-case complexity of the Delaunay diagram is exponential in the number of dimensions, and the convergence of CMA-ES-SoP is relatively slow because of the non-elitist strategy. In this study, we propose an elitist evolutionary algorithm for the optimization on SoP. The proposed method, (1+1)-EA-SoP, adaptively switches two mutation methods; the neighboring-point mutation selects the mutated point from the neighbors on the graph, and the global mutation randomly selects one point. In addition, we develop a novel graph structure that can be constructed with polynomial complexity and possesses several desirable properties related to the Delaunay diagram. The experimental results show that (1+1)-EA-SoP with the proposed graph realizes an effective optimization on SoP.

High-dimensional optimization problems in real-world applications often possess the property called low effective dimensionality (LED), where only a small part of directions in search space affect the evaluation value, and others are redundant. On problems with LED, because the redundant directions deteriorate the prediction performance of the surrogate model, the performance of several surrogate-assisted evolutionary algorithms is worsened. This paper focuses on the doubly trained surrogate CMA-ES (DTS-CMA-ES) that employs Gaussian process regression as a surrogate model and proposes DTS-CMA-ES-LED by incorporating several countermeasures for LED to DTS-CMA-ES. The proposed method considers directions along the eigenvectors of the covariance matrix and evaluates the effectiveness of each direction using the estimated element-wise signal-to-noise ratio of the update directions. Then, the proposed method reconstructs the kernel function with the computed effectiveness to reduce the effect of redundant directions. We also introduce the hyperparameter adaptation mechanism and refinement of the step-size adaptation as countermeasures for LED. The experimental results show that DTS-CMA-ES-LED effectively optimized the benchmark functions with LED.

Bulldozer automation is becoming increasingly important to address the shortage of skilled operators and to improve safety in construction and mining operations. Self-Localization is a key requirement for achieving such automation. However, ensuring reliable self-localization in environments where Global Navigation Satellite System (GNSS) signals are frequently unavailable remains a major challenge. While machine learning-based self-localization methods have gained significant attention, their inherent prediction uncertainties must be accounted for in practical applications. This paper presents a novel self-localization system that relies solely on internal sensors and accounts for both epistemic and aleatoric uncertainties in machine learning predictions. The proposed method first employs a machine learning model to estimate local velocity and its uncertainty, using Deep Ensembles and Gaussian Maximum Likelihood Training. These estimates are then integrated by an Extended Kalman Filter to determine global position. We evaluated our approach using real-world data from an actual bulldozer across diverse operating conditions, such as slope traversal and slalom maneuvers. Experimental results show that our uncertainty-aware method produces more plausible and reliable position estimates than baseline methods that do not account for uncertainty. The proposed system offers a cost-effective solution for enhancing construction equipment autonomy in GNSS-denied environments.

The majority of theoretical analyses of evolutionary algorithms in the discrete domain focus on binary optimization algorithms, even though black-box optimization on the categorical domain has many practical applications. In this paper, we consider a probabilistic model-based algorithm using the family of categorical distributions as its underlying distribution and set the sample size as two. We term this specific algorithm the categorical compact genetic algorithm (ccGA). The ccGA can be considered as an extension of the compact genetic algorithm (cGA), which is an efficient binary optimization algorithm. We theoretically analyze the dependency of the number of possible categories K, the number of dimensions D, and the learning rate η on the runtime. We investigate the tail bound of the runtime on two typical linear functions on the categorical domain: categorical OneMax (COM) and KVal. We derive that the runtimes on COM and KVal are O(Dln(DK)/η) and Θ(DlnK/η) with high probability, respectively. Our analysis is a generalization for that of the cGA on the binary domain.

Black-box optimization problems often require simultaneously optimizing different types of variables, such as continuous, integer, and categorical variables. Unlike integer variables, categorical variables do not necessarily have a meaningful order, and the discretization approach of continuous variables does not work well. Although several Bayesian optimization methods can deal with mixed-category black-box optimization (MC-BBO), they suffer from a lack of scalability to high-dimensional problems and internal computational cost. This paper proposes CatCMA, a stochastic optimization method for MC-BBO problems, which employs the joint probability distribution of multivariate Gaussian and categorical distributions as the search distribution. CatCMA updates the parameters of the joint probability distribution in the natural gradient direction. CatCMA also incorporates the acceleration techniques used in the covariance matrix adaptation evolution strategy (CMA-ES) and the stochastic natural gradient method, such as step-size adaptation and learning rate adaptation. In addition, we restrict the ranges of the categorical distribution parameters by margin to prevent premature convergence and analytically derive a promising margin setting. Numerical experiments show that the performance of CatCMA is superior and more robust to problem dimensions compared to state-of-the-art Bayesian optimization algorithms.

In several real-world applications in medical and control engineering, there are unsafe solutions whose evaluations involve inherent risk. This optimization setting is known as safe optimization and formulated as a specialized type of constrained optimization problem with constraints for safety functions. Safe optimization requires performing efficient optimization without evaluating unsafe solutions. A few studies have proposed the optimization methods for safe optimization based on Bayesian optimization and the evolutionary algorithm. However, Bayesian optimization-based methods often struggle to achieve superior solutions, and the evolutionary algorithm-based method fails to effectively reduce unsafe evaluations. This study focuses on CMA-ES as an efficient evolutionary algorithm and proposes an optimization method termed safe CMA-ES. The safe CMA-ES is designed to achieve both safety and efficiency in safe optimization. The safe CMA-ES estimates the Lipschitz constants of safety functions transformed with the distribution parameters using the maximum norm of the gradient in Gaussian process regression. Subsequently, the safe CMA-ES projects the samples to the nearest point in the safe region constructed with the estimated Lipschitz constants. The numerical simulation using the benchmark functions shows that the safe CMA-ES successfully performs optimization, suppressing the unsafe evaluations, while the existing methods struggle to significantly reduce the unsafe evaluations.

The covariance matrix adaptation evolution strategy (CMA-ES) is a powerful optimization method for continuous black-box optimization problems. Several noise-handling methods have been proposed to bring out the optimization performance of the CMA-ES on noisy objective functions. The adaptations of the population size and the learning rate are two major approaches that perform well under additive Gaussian noise. The reevaluation technique is another technique that evaluates each solution multiple times. In this paper, we discuss the difference between those methods from the perspective of stochastic relaxation that considers the maximization of the expected utility function. We derive that the set of maximizers of the noise-independent utility, which is used in the reevaluation technique, certainly contains the optimal solution, while the noise-dependent utility, which is used in the population size and leaning rate adaptations, does not satisfy it under multiplicative noise. Based on the discussion, we develop the reevaluation adaptation CMA-ES (RA-CMA-ES), which computes two update directions using half of the evaluations and adapts the number of reevaluations based on the estimated correlation of those two update directions. The numerical simulation shows that the RA-CMA-ES outperforms the comparative method under multiplicative noise, maintaining competitive performance under additive noise.

A neural architecture search (NAS) automatically designs the structure of a deep neural network in an exploratory manner that is conventionally designed by experts. A weight-sharing (WS)-based NAS is a time-efficient NAS approach because it simultaneously learns the neural network structure and weight parameters in a single training session. However, WS-based NAS has been observed to have a problem in that the search may converge to an architecture with a significantly low final performance, depending on the design of the search space, which is a set of possible combinations of operations. The design of an appropriate search space is task-dependent, and the user must carefully design it, hindering WS-based NAS applications. We conducted a simple case study on a synthetic regression task to analytically investigate the impact of statistics between operations in the search space on the optimal weight values. Based on the case study, we hypothesized that a strong negative covariance may lead to suboptimal weight values in the following layers, resulting in the selection of a suboptimal architecture. Our numerical experiments show that a simple modification to the aggregation operation in the search space helps mitigate the issue mentioned above and improves the robustness of WS-based NAS.

In real-world applications, a machine learning model is required to handle an open-set recognition (OSR), where unknown classes appear during the inference, in addition to a domain shift, where the data distribution differs between the training and inference phases. Domain generalization (DG) aims to handle the domain shift situation where the target domain of the inference phase is inaccessible during the model training. Open domain generalization (ODG) considers DG and OSR. Domain-augmented meta-learning (DAML) is a method targeting ODG; however, it has a complicated learning process. By contrast, although various DG methods have been proposed, they have not been evaluated in ODG situations. In this study, we comprehensively evaluate the existing DG methods in ODG and show that the two simple DG methods, CORrelation ALignment (CORAL) and maximum mean discrepancy (MMD), are competitive with DAML in several cases. In addition, we propose simple extensions of CORAL and MMD by introducing the techniques used in DAML, such as ensemble learning and Dirichlet mixup data augmentation. The experimental evaluation demonstrates that the extended CORAL and MMD can perform comparably to DAML with lower computational costs. This suggests that the simple DG methods and their simple extensions are strong baselines for ODG.

In the practical application of machine learning, the opaqueness of models often poses significant challenges. While decision trees are known for balancing representability with interpretability, enabling humans to understand decision rules, their interpretability decreases as the complexity of the task increases and the tree size expands, making it difficult to trace and interpret the decision flow. In this paper, we introduce a new variant of decision tree called Bivariate Nonlinear Node Decision Tree (BINN-DT), designed to enhance the interpretability of decision trees. BINN-DT selects bivariate features at each node and utilizes nonlinear splitters to learn the data splitting rules. Additionally, each node visualizes the relationship between the data distribution and split boundaries through a two-dimensional map using the selected bivariate features. Our experiments compared the proposed BINN-DT method with traditional univariate decision trees. The results demonstrate that our approach not only maintains classification accuracy but also produces more compact models. BINN-DT clearly depicts the entire decision boundaries of a model as a tree-structured collection of two-dimensional maps with the bivariate feature axes selected from the entire features. Our method significantly improves the interpretability of models by producing more compact models than the traditional decision trees, without sacrifice of accuracy.

Motivated by the high prediction performance of convolutional neural networks (CNNs), several works have applied them to tabular datasets. As CNNs are built to accept images, several transformations of tabular data have been proposed to obtain images. However, existing methods transform the tabular data into images prior to CNN training, which fails to take the prediction error into account. Additionally, they employ all features from the tables, including unimportant ones, to produce the images. Moreover, the created images might not become human-interpretable because they do not consider the interpretability of images as a metric. To overcome these problems, we propose a hard attention-based converter combined with a convolutional neural network (HACNet), consisting of an attention-based table-to-image converter and a CNN-based predictor. HACNet trains its components simultaneously by minimizing CNN prediction loss and mean squared error (MSE) between created and template images. Minimizing this MSE loss allows us to visually distinguish the created images with different labels. The attention-based converter selects exactly one feature for each pixel in the image via its hard attention mechanism with Gumbel-Softmax, enabling feature selection. We experimentally show that HACNet produces human-interpretable images, reduces used features, and achieves prediction performances comparative with existing methods on several benchmark datasets.

This study targets the mixed-integer black-box optimization (MI-BBO) problem where continuous and integer variables should be optimized simultaneously. The covariance matrix adaptation evolution strategy (CMA-ES), our focus in this study, is a population-based stochastic search method that samples solution candidates from a multivariate Gaussian distribution (MGD), which shows excellent performance in continuous black-box optimization. The parameters of MGD, mean and (co)variance, are updated based on the evaluation value of candidate solutions in the CMA-ES. If the CMA-ES is applied to the MI-BBO with straightforward discretization, however, the variance corresponding to the integer variables becomes much smaller than the granularity of the discretization before reaching the optimal solution, which leads to the stagnation of the optimization. In particular, when binary variables are included in the problem, this stagnation more likely occurs because the granularity of the discretization becomes wider, and the existing integer handling for the CMA-ES does not address this stagnation. To overcome these limitations, we propose a simple integer handling for the CMA-ES based on lower-bounding the marginal probabilities associated with the generation of integer variables in the MGD. The numerical experiments on the MI-BBO benchmark problems demonstrate the efficiency and robustness of the proposed method. Furthermore, to demonstrate the generality of the idea of the proposed method, in addition to the single-objective optimization case, we incorporate it into multi-objective CMA-ES and verify its performance on bi-objective mixed-integer benchmark problems.

The covariance matrix adaptation evolution strategy (CMA-ES) is an efficient continuous black-box optimization method. The CMA-ES possesses many attractive features, including invariance properties and a well-tuned default hyperparameter setting. Moreover, several components to specialize the CMA-ES have been proposed, such as noise handling and constraint handling. To utilize these advantages in mixed-integer optimization problems, the CMA-ES with margin has been proposed. The CMA-ES with margin prevents the premature convergence of discrete variables by the margin correction, in which the distribution parameters are modified to leave the generation probability for changing the discrete variable. The margin correction has been applied to (μ/μw,Λ)-CMA-ES, while this paper introduces the margin correction into (1+1)-CMA-ES, an elitist version of CMA-ES. The (1+1)-CMA-ES is often advantageous for unimodal functions and can be computationally less expensive. To tackle the performance deterioration on mixed-integer optimization, we use the discretized elitist solution as the mean of the sampling distribution and modify the margin correction not to move the elitist solution. The numerical simulation using benchmark functions on mixed-integer, integer, and binary domains shows that (1+1)-CMA-ES with margin outperforms the CMA-ES with margin and is better than or comparable with several specialized methods to a particular search domain.

Neural architecture search (NAS) is a framework for automating the design process of a neural network structure. While the recent one-shot approaches have reduced the search cost, there still exists an inherent trade-off between cost and performance. It is important to appropriately stop the search and further reduce the high cost of NAS. Meanwhile, the differentiable architecture search (DARTS), a typical one-shot approach, is known to suffer from overfitting. Heuristic early-stopping strategies have been proposed to overcome such performance degradation. In this paper, we propose a more versatile and principled early-stopping criterion on the basis of the evaluation of a gap between expectation values of generalisation errors of the previous and current search steps with respect to the architecture parameters. The stopping threshold is automatically determined at each search epoch without cost. In numerical experiments, we demonstrate the effectiveness of the proposed method. We stop the one-shot NAS algorithms and evaluate the acquired architectures on the benchmark datasets: NAS-Bench-201 and NATS-Bench. Our algorithm is shown to reduce the cost of the search process while maintaining a high performance.

This study targets the mixed-integer black-box optimization (MI-BBO) problem where continuous and integer variables should be optimized simultaneously. The CMA-ES, our focus in this study, is a population-based stochastic search method that samples solution candidates from a multivariate Gaussian distribution (MGD), which shows excellent performance in continuous BBO. The parameters of MGD, mean and (co)variance, are updated based on the evaluation value of candidate solutions in the CMA-ES. If the CMA-ES is applied to the MI-BBO with straightforward discretization, however, the variance corresponding to the integer variables becomes much smaller than the granularity of the discretization before reaching the optimal solution, which leads to the stagnation of the optimization. In particular, when binary variables are included in the problem, this stagnation more likely occurs because the granularity of the discretization becomes wider, and the existing modification to the CMA-ES does not address this stagnation. To overcome these limitations, we propose a simple modification of the CMA-ES based on lower-bounding the marginal probabilities associated with the generation of integer variables in the MGD. The numerical experiments on the MI-BBO benchmark problems demonstrate the efficiency and robustness of the proposed method.

This paper proposes a two-phase framework with a Bézier simplex-based interpolation method (TPB) for computationally expensive multi-objective optimization. The first phase in TPB aims to approximate a few Pareto optimal solutions by optimizing a sequence of single-objective scalar problems. The first phase in TPB can fully exploit a state-of-the-art single-objective derivative-free optimizer. The second phase in TPB utilizes a Bézier simplex model to interpolate the solutions obtained in the first phase. The second phase in TPB fully exploits the fact that a Bézier simplex model can approximate the Pareto optimal solution set by exploiting its simplex structure when a given problem is simplicial. We investigate the performance of TPB on the 55 bi-objective BBOB problems. The results show that TPB performs significantly better than HMO-CMA-ES and some state-of-the-art meta-model-based optimizers.

Understanding the relationship between material characteristics and microscopic structure is important for the development of solder materials. To clarify this relationship, machine learning approaches are often used to predict material characteristics from microstructural images. Although a trained machine learning model can predict the material characteristics for a given microstructural image, it cannot directly create microstructural images of solder materials with desirable characteristics. Therefore, it is difficult to use machine learning to develop new solder materials. This paper presents a method for generating electron probe micro-analyzer (EPMA) images of a solder with desirable characteristics using deep learning. Our method uses a generative adversarial network (GAN) to generate images, and a convolutional neural network (CNN)-based evaluator to predict their characteristics. A common difficulty in applying machine learning to material science is the lack of training data, which often results in predictions with low accuracy. To address the small dataset problem, we trained the ranking prediction model of the characteristics instead of the regression model. Moreover, we employed transfer learning, in which a CNN model trained on texture datasets was used as the initial model. The experimental results show that the GAN successfully generated EPMA images that were similar to the actual images. The use of the ranking prediction model and transfer learning improved the performance of the CNN-based characteristic evaluator. We then selected promising generated EPMA images using the CNN-based characteristic evaluator and found that the characteristics of the selected EPMA images were consistent with expert experience.

Conversational gestures have a crucial role in realizing natural interactions with virtual agents and robots. Data-driven approaches, such as deep learning and machine learning, are promising in constructing the gesture generation model, which automatically provides the gesture motion for speech or spoken texts. This study experimentally analyzes a deep learning-based gesture generation model from spoken text using a convolutional neural network. The proposed model takes a sequence of spoken words as the input and outputs a sequence of 2D joint coordinates representing the conversational gesture motion. We prepare a dataset consisting of gesture motions and spoken texts by adding text information to an existing dataset and train the models using specific speaker's data. The quality of the generated gestures is compared with those from an existing speech-to-gesture generation model through a user perceptual study. The subjective evaluation shows that the model performance is comparable or superior to those by the existing speech-to-gesture generation model. In addition, we investigate the importance of data cleansing and loss function selection in the text-to-gesture generation model. We further examine the model transferability between speakers. The experimental results demonstrate successful model transferability of the proposed model. Finally, we show that the text-to-gesture generation model can produce good quality gestures even when using a transformer architecture.

The benchmark datasets for neural architecture search (NAS) have been developed to alleviate the computationally expensive evaluation process and ensure a fair comparison. Recent NAS benchmarks only focus on architecture optimization, although the training hyperparameters affect the obtained model performances. Building the benchmark dataset for joint optimization of architecture and training hyperparameters is essential to further NAS research. The existing NAS-HPO-Bench is a benchmark for joint optimization, but it does not consider the network connectivity design as done in modern NAS algorithms. This paper introduces the first benchmark dataset for joint optimization of network connections and training hyperparameters, which we call NAS-HPO-Bench-II. We collect the performance data of 4K cell-based convolutional neural network architectures trained on the CIFAR-10 dataset with different learning rate and batch size settings, resulting in the data of 192K configurations. The dataset includes the exact data for 12 epoch training. We further build the surrogate model predicting the accuracies after 200 epoch training to provide the performance data of longer training epoch. By analyzing NAS-HPO-Bench-II, we confirm the dependency between architecture and training hyperparameters and the necessity of joint optimization. Finally, we demonstrate the benchmarking of the baseline optimization algorithms using NAS-HPO-Bench-II.

Although convolutional neural networks (CNNs) have been widely used in image classification, a large amount of training data is required for training models. Especially in industrial applications, not only is there a demand for reducing the cost of labeling images, but also, it is difficult to obtain a large amount of training images. Several approaches have been proposed to improve classification accuracy by embedding human knowledge other than labels into the models. However, in these approaches, the annotation methods tend to become complex and applicable only to specific model architectures corresponding to those methods.In this study, we propose a new approach to improve the image classification accuracy of CNNs by adding human knowledge as simple region annotations. In our method, annotators draw the regions that attract the most attention as a single ellipse per image for classifying image categories. This annotation is called non-strict attentional region annotation (NARA) and is used for training CNNs together with the labels. We also propose a training method using NARA, which is applicable to common CNNs. For the proof of concept, we have added region annotations to all 5,000 images of the STL-10 training dataset. Through experiments, we demonstrate that our approach is effective to improve image classification accuracy with relatively inexpensive additional cost. Our annotation dataset is available at: https://git.io/stl10nara.

The convolutional neural network (CNN), one of the deep learning models, has demonstrated outstanding performance in a variety of computer vision tasks. However, as the network architectures become deeper and more complex, designing CNN architectures requires more expert knowledge and trial and error. In this article, we attempt to automatically construct high-performing CNN architectures for a given task. Our method uses Cartesian genetic programming (CGP) to encode the CNN architectures, adopting highly functional modules such as a convolutional block and tensor concatenation, as the node functions in CGP. The CNN structure and connectivity, represented by the CGP, are optimized to maximize accuracy using the evolutionary algorithm. We also introduce simple techniques to accelerate the architecture search: rich initialization and early network training termination. We evaluated our method on the CIFAR-10 and CIFAR-100 datasets, achieving competitive performance with state-of-the-art models. Remarkably, our method can find competitive architectures with a reasonable computational cost compared to other automatic design methods that require considerably more computational time and machine resources.

We theoretically analyze the information geometric optimization (IGO), which is a unified framework of stochastic search algorithms for black-box optimization. The IGO framework has two parameters: 1) the learning rate and 2) the sample size, and they influence the behavior of the algorithm. We investigate the strategy parameters of the IGO with the family of isotropic Gaussian distributions on a general convex quadratic function. Compared to the previous theoretical works, where an infinite sample size is assumed and the deterministic algorithm dynamics is studied, we investigate the expected improvement of the algorithm with a finite sample size. The analysis finds that the relative decrease rates of the distance from the distribution mean to the landscape optimum and the distribution standard deviation must be the same, which we observe in practice, while the analysis based on an infinite sample size failed to obtain. We derive these rates explicitly as a function of the eigenvalues of the Hessian of the objective function and the strategy parameters. We also derive the stable value of the ratio of the square distance to the optimum over the distribution variance, as well as the conditions that the stable value exists. These theoretical values coincide with our numerical simulations.

High sensitivity of neural architecture search (NAS) methods against their input such as step-size (i.e., learning rate) and search space prevents practitioners from applying them out-of-the-box to their own problems, albeit its purpose is to automate a part of tuning process. Aiming at a fast, robust, and widely-applicable NAS, we develop a generic optimization framework for NAS. We turn a coupled optimization of connection weights and neural architecture into a differentiable optimization by means of stochastic relaxation. It accepts arbitrary search space (widely-applicable) and enables to employ a gradient-based simultaneous optimization of weights and architecture (fast). We propose a stochastic natural gradient method with an adaptive step-size mechanism built upon our theoretical investigation (robust). Despite its simplicity and no problem-dependent parameter tuning, our method exhibited near state-of-the-art performances with low computational budgets both on image classification and inpainting tasks.

Deep neural networks (DNNs) are powerful machine learning models and have succeeded in various artificial intelligence tasks. Although various architectures and modules for the DNNs have been proposed, selecting and designing the appropriate network structure for a target problem is a challenging task. In this paper, we propose a method to simultaneously optimize the network structure and weight parameters during neural network training. We consider a probability distribution that generates network structures, and optimize the parameters of the distribution instead of directly optimizing the network structure. The proposed method can apply to the various network structure optimization problems under the same framework. We apply the proposed method to several structure optimization problems such as selection of layers, selection of unit types, and selection of connections using the MNIST, CIFAR-10, and CIFAR-100 datasets. The experimental results show that the proposed method can find the appropriate and competitive network structures.

In this article, we theoretically investigate the convergence properties of the information geometric optimization (IGO) algorithm given the family of isotropic Gaussian distributions on the sphere function. Differently from previous studies, where the exact natural gradient is taken, i.e., the infinite samples are assumed, we consider the case that the natural gradient is estimated from finite samples. We derive the rates of the expected decrease of the squared distance to the optimum and the variance parameter as functions of the learning rates, dimension, and sample size. From the rates of decrease deduces that the rates of decreases of the squared distance to the optimum and the variance parameter must agree for geometric convergence of the algorithm. In other words, the ratio between the squared distance to the optimum and the variance must be stable, which is observed empirically but is not derived in the previous theoretical studies. We further derive the condition on the learning rates that the rates of decreases agree and derive the stable value of the ratio. We confirm in simulation that the derived rates of decreases and the stable value of the ratio well approximate the behavior of the IGO algorithm.

We propose a method for designing convolutional neural network (CNN) architectures based on Cartesian genetic programming (CGP). In the proposed method, the architectures of CNNs are represented by directed acyclic graphs, in which each node represents highly-functional modules such as convolutional blocks and tensor operations, and each edge represents the connectivity of layers. The architecture is optimized to maximize the classification accuracy for a validation dataset by an evolutionary algorithm. We show that the proposed method can find competitive CNN architectures compared with state-of-the-art methods on the image classification task using CIFAR-10 and CIFAR-100 datasets.

We present a novel framework to automatically generate natural gesture motions accompanying speech from audio utterances. Based on a Bi-Directional LSTM Network, our deep network learns speech-gesture relationships with both backward and forward consistencies over a long period of time. Our network regresses a full 3D skeletal pose of a human from perceptual features extracted from the input audio in each time step. Then, we apply combined temporal filters to smooth out the generated pose sequences. We utilize a speech-gesture dataset recorded with a headset and marker-based motion capture to train our network. We validated our approach with a subjective evaluation and compared it against "original" human gestures and "mismatched" human gestures taken from a different utterance. The evaluation result shows that our generated gestures are significantly better than the "mismatched" gestures with respect to time consistency. The generated gesture also shows marginally significant improvement in terms of semantic consistency when compared to "mismatched" gestures.

The convolutional neural network (CNN), which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, we attempt to automatically construct CNN architectures for an image classification task based on Cartesian genetic programming (CGP). In our method, we adopt highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The CNN structure and connectivity represented by the CGP encoding method are optimized to maximize the validation accuracy. To evaluate the proposed method, we constructed a CNN architecture for the image classification task with the CIFAR-10 dataset. The experimental result shows that the proposed method can be used to automatically find the competitive CNN architecture compared with state-of-the-art models.

In this research, we take a first step in generating motion data for gestures directly from speech features. Such a method can make creating gesture animations for Embodied Conversational Agents much easier. We implemented a model using Bi-Directional LSTM taking phonemic features from speech audio data as input to output time sequence data of rotations of bone joints. We assessed the validity of the predicted gesture motion data by evaluating the final loss value of the network, and evaluating the impressions of the predicted gesture by comparing it with the actual motion data that accompanied the audio data used for input and motion data that accompanied a different audio data. The results showed that the accuracy of the prediction for the LSTM model was better than a simple RNN model. In contrast, the impressions evaluation of the predicted gesture was rated lower than the original and mismatched gestures, although individually some predicted gestures were rated the same degree as the mismatched gestures.

Bayesian optimization is a black-box optimization method, which maintains a surrogate model learned by using previously evaluated solutions and selects the next solution to be evaluated using the model. Since the Bayesian optimization method models the objective function as it is, it is not invariant under the order preserving transformation of the objective function. On the other hand, many evolutionary algorithms have that property only by using the ranking information of solutions. In this paper, we introduce two types of invariant objective function: the ranking-based objective and the Lebesgue measure-based objective, into the Bayesian optimization in order to realize the invariance property. The impact of the invariant objective function for the search performance is verified through the numerical experiment. The experimental result shows that the introduced objectives achieve the invariance for the order preserving transformation without the considerable performance deterioration in the Bayesian optimization.

In this paper, we design a hierarchical feature construction method for image classification. Our method has two feature construction stages: (1) feature construction by a combination of primitive image processing filters, and (2) feature construction by evolved filters. We verify the image classification performance of the proposed method on the MIT urban and nature scene dataset. The experimental results show that the two-stage feature construction improves the classification accuracy compared to single stage feature construction. In addition, the proposed method outperforms several existing feature construction methods.

Recent studies reveal that the covariance matrix adaptation evolution strategy (CMA-ES) updates the parameters based on the natural gradient. The rank-based weight is considered the result of the quantile-based transformation of the objective value and the parameters are adjusted in the direction of the natural gradient estimated by Monte-Carlo with the samples drawn from the current distribution. In this paper, we propose a sample reuse mechanism for the CMA-ES. On the basis of the importance sampling, the past samples are reused to reduce the estimation variance of the quantile and the natural gradient. We derive the formula for the rank-¥mu update of the covariance matrix and the mean vector update using the past samples, then incorporate it into the CMA-ES without the step-size adaptation. From the numerical experiments, we observe that the proposed approach helps to reduce the number of function evaluations on many benchmark functions, especially when the number of samples at each iteration is relatively small.

One of the fast similarity search techniques is a binary hashing method that transforms a real-valued vector into a binary code. The similarity between two binary codes is measured by their Hamming distance. In this method, a hash table is often used for realizing the constant time similarity search. The number of accesses to the hash table, however, increases when the number of bits becomes long. In this paper, we consider the method that does not access the data with long Hamming radius by using multiple binary codes. Then, we propose the learning method of the binary hash functions for multiple binary codes. We conduct the experiment on similarity search utilizing up to 20 million data set, and show that our proposed method achieves a fast similarity search compared with the conventional linear scan and hash table search.

Image processing and recognition technologies are becoming increasingly important. Automatic construction methods for image transformation algorithms proposed to date approximate adequate image transformation from original images to their target images using a combination of several known image processing filters by evolutionary computation techniques. In this paper, we introduce the adaptive image processing filters that process according to the features of an input image. The processing of the adaptive filters is decided based on the local features of an input image. We implement them to feed-forward genetic image network (FFGIN) that is one of the automatic construction methods for image transformations. Then we apply our method to the problems of segmentation of organs and tissues in medical images. Experimental results show that our method constructs the effective segmentation algorithms that extract multiple regions respectively.

When we evaluate the search performance of an evolutionary computation (EC) technique, we usually apply it to typical benchmark functions and evaluate its performance in comparison to other techniques. In experiments on limited benchmark functions, it can be difficult to understand the features of each technique. In this paper, the search spaces that emphasize the performance difference of EC techniques are evolved by Cartesian genetic programming. We focus on a real-coded genetic algorithm, which is a type of genetic algorithm that has a real-valued vector as a chromosome. In particular, we generate search spaces using the performance difference of real-coded crossovers. In the experiments, we evolve the search spaces using the combination of three types of real-coded crossovers. As a result of our experiments, the search spaces that exhibit the largest performance difference of two crossovers are generated for all the combinations.

In the fields of image processing and recognition, image segmentation is an important basic technique in which an image is partitioned into multiple regions (sets of pixels). In this paper, we propose a method for evolutionary image segmentation based on multiobjective clustering. In this method, two objectives, overall deviation and edge value, are optimized simultaneously using a multiobjective evolutionary algorithm. These objectives are important factors for image segmentation. The proposed method finds various solutions (image segmentation results) by the use of an evolutionary process. We apply the proposed method to several image segmentation problems and confirm that various solutions are obtained. In addition, we use a simple heuristic method to select one solution from the original Pareto solutions and show that a good image segmentation result is selected.

Currently, various automatic programming techniques have been proposed and applied in various fields. Graph Structured Program Evolution (GRAPE) is a recent automatic programming technique with graph structure. This technique can generate complex programs automatically. In this paper, we introduce the concept of automatically defined functions, called automatically defined nodes (ADN), in GRAPE. The proposed GRAPE program has a main program and several subprograms. We verified the effectiveness of ADN through several program evolution experiments, and report the results of evolution of recursive programs using GRAPE modified with ADN.

In recent years a lot of Automatic Programming techniques have developed. A typical example of Automatic Programming is Genetic Programming (GP), and various extensions and representations for GP have been proposed so far. However, it seems that more improvements are necessary to obtain complex programs automatically. In this paper we proposed a new method called Graph Structured Program Evolution (GRAPE). The representation of GRAPE is graph structure, therefore it can represent complex programs (e.g. branches and loops) using its graph structure. Each program is constructed as an arbitrary directed graph of nodes and data set. The GRAPE program handles multiple data types using the data set for each type, and the genotype of GRAPE is the form of a linear string of integers. We apply GRAPE to four test problems, factorial, Fibonacci sequence, exponentiation and reversing a list, and demonstrate that the optimum solution in each problem is obtained by the GRAPE system.

In this paper, we apply graph structured program evolution (GRAPE) to evolution of general sorting algorithm. GRAPE is a new Automatic Programming technique. The representation of GRAPE is graph structure, therefore it can express complex programs (e.g. branches and loops) using its graph structure. Each program is constructed as an arbitrary directed graph of nodes and data set. GRAPE handles multiple data types using data set for each type, and the genotype of GRAPE is the form of a linear string of integers. The aim of this work is to evolve a program which correctly sort any sequence of numbers. We demonstrate that GRAPE constructs general sorting algorithm automatically.