Pareto-optimal sets of multiobjective optimization problems with black-box and computationally expensive objective functions are generally hard to locate within a limited computational budget, and this situation gets even worse when more than three objectives are involved. To this end, we present a novel surrogate-assisted many-objective optimization algorithm RECAS. Unlike most prior studies, the proposed algorithm is a non–evolutionary-based method, and it iteratively determines new points for expensive evaluation via a series of independent reference vector assisted candidate searches. Furthermore, to make the number of surrogates to be maintained independent of the number of objectives, in each candidate search, RECAS constructs a surrogate model in an aggregated manner to approximate the quality assessment indicator of each point rather than a certain objective function. Under some mild assumptions, this study proves that RECAS converges almost surely to the Pareto-optimal front. In the numerical experiments, the effectiveness and reliability of RECAS are examined on both DTLZ and WFG test suites with the number of objectives varying from 2 to 10. Compared with six state-of-the-art many-objective optimization algorithms, RECAS generally performs better in maintaining convergent and well-spread approximation of the Pareto-optimal front. Finally, the good performance of RECAS on two watershed simulation model calibration problems indicates its great potential in handling real-world applications. History: Accepted by Antonio Frangioni, Area Editor for Design and Analysis of Algorithms–Continuous. Funding: This work was supported by the National University of Singapore start-up grant to C.A. Shoemaker [Grant R-266-000-109-133]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplementary Information [ https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1260 ] or is available from the IJOC GitHub software repository ( https://github.com/INFORMSJoC ) at [ http://dx.doi.org/10.5281/zenodo.7243971 ].

To improve deep neural net hyper-parameter optimization we develop a deterministic surrogate optimization algorithm as an efficient alternative to Bayesian optimization. A deterministic Radial Basis Function (RBF) surrogate model is built to interpolate previously evaluated points, and this surrogate model is incrementally updated in each iteration. The stochastic algorithm CMA-ES is used to search the acquisition function based on the surrogate. The acquisition function at a point is based on a weighted average of the surrogate at x and the minimum distance from x to a previously evaluated point. We evaluate the proposed algorithm RBF-CMA on hyper-parameter optimization tasks for deep convolutional neural networks on datasets of CIFAR-10, SVHN, and CIFAR-100. We show that RBF-CMA achieves a promising performance especially when the search space dimension is high in comparison to other algorithms including GP-EI, GP-LCB, and SMBO.

Multi-tenancy for latency-critical applications leads to resource interference and unpredictable performance. Core reconfiguration opens up more opportunities for application colocation, as it allows the hardware to adjust to the dynamic performance and power needs of a specific mix of co-scheduled services. However, reconfigurability also introduces challenges, as even for a small number of reconfigurable cores, exploring the design space becomes more time- and resource-demanding.We present CuttleSys, a runtime for reconfigurable multicores that leverages scalable and lightweight data mining to quickly identify suitable core and cache configurations for a set of co-scheduled applications. The runtime combines collaborative filtering to infer the behavior of each job on every core and cache configuration, with Dynamically Dimensioned Search to efficiently explore the configuration space. We evaluate CuttleSys on multicores with tens of reconfigurable cores and show up to 2.46× and 1.55× performance improvements compared to core-level gating and oracle-like asymmetric multicores respectively, under stringent power constraints.

Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like Gaussian processes to approximate and minimize the validation error function of hyperparameter values. However, probabilistic surrogates require accurate estimates of sufficient statistics (e.g., covariance) of the error distribution and thus need many function evaluations with a sizeable number of hyperparameters. This makes them inefficient for optimizing hyperparameters of deep learning algorithms, which are highly expensive to evaluate. In this work, we propose a new deterministic and efficient hyperparameter optimization method that employs radial basis functions as error surrogates. The proposed mixed integer algorithm, called HORD, searches the surrogate for the most promising hyperparameter values through dynamic coordinate search and requires many fewer function evaluations. HORD does well in low dimensions but it is exceptionally better in higher dimensions. Extensive evaluations on MNIST and CIFAR-10 for four deep neural networks demonstrate HORD significantly outperforms the well-established Bayesian optimization methods such as GP, SMAC, and TPE. For instance, on average, HORD is more than 6 times faster than GP-EI in obtaining the best configuration of 19 hyperparameters.

GOMORS is a parallel response surface-assisted evolutionary algorithm approach to multi-objective optimization that is designed to obtain good non-dominated solutions to black box problems with relatively few objective function evaluations. GOMORS uses Radial Basic Functions to iteratively compute surrogate response surfaces as an approximation of the computationally expensive objective function. A multi objective search utilizing evolution, local search, multi method search and non-dominated sorting is done on the surrogate radial basis function surface because it is inexpensive to compute. A balance between exploration, exploitation and diversification is obtained through a novel procedure that simultaneously selects evaluation points within an algorithm iteration through different metrics including Approximate Hypervolume Improvement, Maximizing minimum domain distance, Maximizing minimum objective space distance, and surrogate-assisted local search, which can be computed in parallel. The results are compared to ParEGO (a kriging surrogate method solving many weighted single objective optimizations) and the widely used NSGA-II. The results indicate that GOMORS outperforms ParEGO and NSGA-II on problems tested. For example, on a groundwater PDE problem, GOMORS outperforms ParEGO with 100, 200 and 400 evaluations for a 6 dimensional problem, a 12 dimensional problem and a 24 dimensional problem. For a fixed number of evaluations, the differences in performance between GOMORS and ParEGO become larger as the number of dimensions increase. As the number of evaluations increase, the differences between GOMORS and ParEGO become smaller. Both surrogate-based methods are much better than NSGA-II for all cases considered.

This paper presents a parallel surrogate-based global optimization method for computationally expensive objective functions that is more effective for larger numbers of processors. To reach this goal, we integrated concepts from multi-objective optimization and tabu search into, single objective, surrogate optimization. Our proposed derivative-free algorithm, called SOP, uses non-dominated sorting of points for which the expensive function has been previously evaluated. The two objectives are the expensive function value of the point and the minimum distance of the point to previously evaluated points. Based on the results of non-dominated sorting, P points from the sorted fronts are selected as centers from which many candidate points are generated by random perturbations. Based on surrogate approximation, the best candidate point is subsequently selected for expensive evaluation for each of the P centers, with simultaneous computation on P processors. Centers that previously did not generate good solutions are tabu with a given tenure. We show almost sure convergence of this algorithm under some conditions. The performance of SOP is compared with two RBF based methods. The test results show that SOP is an efficient method that can reduce time required to find a good near optimal solution. In a number of cases the efficiency of SOP is so good that SOP with 8 processors found an accurate answer in less wall-clock time than the other algorithms did with 32 processors.

SO-MODS is a new algorithm that combines surrogate global optimization methods with local search. SO-MODS is an extension of prior algorithms that sought to find near optimal solutions for computationally very expensive functions for which the number of allowable evaluations is strictly limited. The global search method in SO-MODS perturbs the best point found so far in order to find a new sample point. The number of decision variables being perturbed is dynamically adjusted in each iteration in order to be more effective for higher dimensional problems. The procedure for dynamically changing the dimensions perturbed is drawn from earlier work on the DYCORS algorithm. We use a cubic radial basis function as surrogate model and investigate two approaches to improve the solution accuracy. The numerical results show that SO-MODS is able to reduce the objective function value dramatically with just a few hundred evaluations even for 30-dimensional problems. The local search is then able to reduce the objective function value further.

Future microprocessors may become so power constrained that not all transistors will be able to be powered on at once. These systems will be required to nimbly adapt to changes in the chip power that is allocated to general-purpose cores and to specialized accelerators.

Future many-core microprocessors are likely to be heterogeneous, by design or due to variability and defects. The latter type of heterogeneity is especially challenging due to its unpredictability. To minimize the performance and power impact of these hardware imperfections, the runtime thread scheduler and global power manager must be nimble enough to handle such random heterogeneity. With hundreds of cores expected on a single die in the future, these algorithms must provide high power-performance efficiency, yet remain scalable with low runtime overhead.

We develop a parallel implementation of a stochastic radial basis function (RBF) algorithm for global optimization by Regis and Shoemaker [Regis, R. G., C. A. Shoemaker. 2007a. A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4) 497–509]. The proposed parallel algorithm is suitable for the global optimization of computationally expensive objective functions and does not require derivatives. Each iteration of the algorithm consists of building an RBF model to approximate the expensive function and using this model to select multiple points for simultaneous function evaluation on multiple processors. The function evaluation points are selected from a set of random candidate points according to two criteria: estimated function value based on the RBF model, and minimum distance from previously evaluated points and previously selected points within each iteration. We compare the performance of our parallel stochastic RBF algorithm against alternative parallel global optimization methods, including two multistart parallel finite-difference quasi-Newton methods, a multistart implementation of Asynchronous Parallel Pattern Search [Hough, P., T. G. Kolda, V. J. Torczon. 2001. Asynchronous parallel pattern search for nonlinear optimization. SIAM J. Sci. Comput. 23(1) 134–156], a parallel implementation of Probabilistic Global Search Lausanne [Raphael, B., I. F. C. Smith. 2003. A direct stochastic algorithm for global search. Appl. Math. Comput. 146 729–758], a parallel evolutionary algorithm, and a deterministic parallel RBF algorithm by Regis and Shoemaker [Regis, R. G., C. A. Shoemaker. 2007c. Parallel radial basis function methods for the global optimization of expensive functions. Eur. J. Oper. Res. 182(2) 514–535]. We report good results for our parallel stochastic RBF method when using one, four, or eight processors in comparison with the alternatives on 20 test problems and on 3 optimization problems involving groundwater bioremediation.

We introduce a new framework for the global optimization of computationally expensive multimodal functions when derivatives are unavailable. The proposed Stochastic Response Surface (SRS) Method iteratively utilizes a response surface model to approximate the expensive function and identifies a promising point for function evaluation from a set of randomly generated points, called candidate points. Assuming some mild technical conditions, SRS converges to the global minimum in a probabilistic sense. We also propose Metric SRS (MSRS), which is a special case of SRS where the function evaluation point in each iteration is chosen to be the best candidate point according to two criteria: the estimated function value obtained from the response surface model, and the minimum distance from previously evaluated points. We develop a global optimization version and a multistart local optimization version of MSRS. In the numerical experiments, we used a radial basis function (RBF) model for MSRS and the resulting algorithms, Global MSRBF and Multistart Local MSRBF, were compared to 6 alternative global optimization methods, including a multistart derivative-based local optimization method. Multiple trials of all algorithms were compared on 17 multimodal test problems and on a 12-dimensional groundwater bioremediation application involving partial differential equations. The results indicate that Multistart Local MSRBF is the best on most of the higher dimensional problems, including the groundwater problem. It is also at least as good as the other algorithms on most of the lower dimensional problems. Global MSRBF is competitive with the other alternatives on most of the lower dimensional test problems and also on the groundwater problem. These results suggest that MSRBF is a promising approach for the global optimization of expensive functions.

We develop an approach for the optimization of continuous costly functions that uses a space-filling experimental design and local function approximation to reduce the number of function evaluations in an evolutionary algorithm. Our approach is to estimate the objective function value of an offspring by fitting a function approximation model over the k nearest previously evaluated points, where k=(d+1)(d+2)/2 and d is the dimension of the problem. The estimated function values are used to screen offspring to identify the most promising ones for function evaluation. To fit function approximation models, a symmetric Latin hypercube design (SLHD) is used to determine initial points for function evaluation. We compared the performance of an evolution strategy (ES) with local quadratic approximation, an ES with local cubic radial basis function (RBF) interpolation, an ES whose initial parent population comes from an SLHD, and a conventional ES. These algorithms were applied to a twelve-dimensional (12-D) groundwater bioremediation problem involving a complex nonlinear finite-element simulation model. The performances of these algorithms were also compared on the Dixon-Szego test functions and on the ten-dimensional (10-D) Rastrigin and Ackley test functions. All comparisons involve analysis of variance (ANOVA) and the computation of simultaneous confidence intervals. The results indicate that ES algorithms with local approximation were significantly better than conventional ES algorithms and ES algorithms initialized by SLHDs on all Dixon-Szego test functions except for Goldstein-Price. However, for the more difficult 10-D and 12-D functions, only the cubic RBF approach was successful in improving the performance of an ES. Moreover, the results also suggest that the cubic RBF approach is superior to the quadratic approximation approach on all test functions and the difference in performance is statistically significant for all test functions with dimension d/spl ges/4.

This paper describes a new parallel algorithm for optimizing costly nonconvex functions with box constraints when derivatives are unavailable. MAPO (Multi-Algorithm Parallel Optimization) iteratively uses a committee of optimization algorithms based on response surfaces to generate candidate points for function evaluation. In each iteration, the evaluation points are selected from multiple rankings of all candidate points. Good numerical results for MAPO with 4 radial basis function methods are reported for 8 processors.

We compare two partitioning methods for solving a multi-dimensional optimal control problem in parallel using continuous state, continuous control, stochastic dynamic programming (SCP). The algorithm uses tensor products to interpolate the multi-dimensional, continuous variable state space. Unlike previous parallel SDP applications, an iterative optimization routine is used to find the optimal continuous control for each node point in the state space. This routine causes load balance and dependence problems not encountered in previous parallel SDP applications. Even with these additional computational complexities, superb efficiencies are achieved on a shared memory MIMD architecture. The two methods of solution differ in the following way: Method 1 uses the outer state loop index to send subsets of the state space (called parcels) to each concurrent process. The number of parcels sent as parallel tasks is the number of discretizations of one of the state variables. Method 2 allows the user to determine the number of parcels to be sent as parallel tasks. Method 2 splits the state space into a greater number of parcels, so that the processors are more likely to finish at the same time at a synchronization barrier. Computational results indicate that Method 2 has better load balance, with efficiencies up to 93.5%. INFORMS Journal on Computing, ISSN 1091-9856, was published as ORSA Journal on Computing from 1989 to 1995 under ISSN 0899-1499.