Offline multi-objective optimization aims to identify Pareto-optimal solutions given a dataset of designs and their objective values. In this work, we propose a preference-guided diffusion model that generates Pareto-optimal designs by leveraging a classifier-based guidance mechanism. Our guidance classifier is a preference model trained to predict the probability that one design dominates another, directing the diffusion model toward optimal regions of the design space. Crucially, this preference model generalizes beyond the training distribution, enabling the discovery of Pareto-optimal solutions outside the observed dataset. We introduce a novel diversity-aware preference guidance, augmenting Pareto dominance preference with diversity criteria. This ensures that generated solutions are optimal and well-distributed across the objective space, a capability absent in prior generative methods for offline multi-objective optimization. We evaluate our approach on various continuous offline multi-objective optimization tasks and find that it consistently outperforms other inverse/generative approaches while remaining competitive with forward/ surrogate-based optimization methods. Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions that approximate the Pareto front well.

We consider the problem of multi-objective optimization (MOO) of expensive black-box functions with the goal of discovering high-quality and diverse Pareto fronts where we are allowed to evaluate a batch of inputs. This problem arises in many real-world applications including penicillin production where diversity of solutions is critical. We solve this problem in the framework of Bayesian optimization (BO) and propose a novel approach referred to as Pareto front-Diverse Batch Multi-Objective BO (PDBO). PDBO tackles two important challenges: 1) How to automatically select the best acquisition function in each BO iteration, and 2) How to select a diverse batch of inputs by considering multiple objectives. We propose principled solutions to address these two challenges. First, PDBO employs a multi-armed bandit approach to select one acquisition function from a given library. We solve a cheap MOO problem by assigning the selected acquisition function for each expensive objective function to obtain a candidate set of inputs for evaluation. Second, it utilizes Determinantal Point Processes (DPPs) to choose a Pareto-front-diverse batch of inputs for evaluation from the candidate set obtained from the first step. The key parameters for the methods behind these two steps are updated after each round of function evaluations. Experiments on multiple MOO benchmarks demonstrate that PDBO outperforms prior methods in terms of both the quality and diversity of Pareto solutions.

We consider the problem of constrained multi-objective optimization over black-box objectives, with user-defined preferences, with a largely infeasible input space. Our goal is to approximate the optimal Pareto set from the small fraction of feasible inputs. The main challenges include huge design space, multiple objectives, numerous constraints, and rare feasible inputs identified only through expensive experiments. We present PAC-MOO, a novel preference-aware multi-objective Bayesian optimization algorithm to solve this problem. It leverages surrogate models for objectives and constraints to intelligently select the sequence of inputs for evaluation to achieve the target goal.

We consider the problem of active learning for global sensitivity analysis of expensive black-box functions. Our aim is to efficiently learn the importance of different input variables, e.g., in vehicle safety experimentation, we study the impact of the thickness of various components on safety objectives. Since function evaluations are expensive, we use active learning to prioritize experimental resources where they yield the most value. We propose novel active learning acquisition functions that directly target key quantities of derivative-based global sensitivity measures (DGSMs) under Gaussian process surrogate models. We showcase the first application of active learning directly to DGSMs, and develop tractable uncertainty reduction and information gain acquisition functions for these measures. Through comprehensive evaluation on synthetic and real-world problems, our study demonstrates how these active learning acquisition strategies substantially enhance the sample efficiency of DGSM estimation, particularly with limited evaluation budgets. Our work paves the way for more efficient and accurate sensitivity analysis in various scientific and engineering applications.

The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as Budget-Aware Planning for Iterative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.

Optimizing expensive to evaluate black-box functions over an input space consisting of all permutations of d objects is an important problem with many real-world applications. For example, placement of functional blocks in hardware design to optimize performance via simulations. The overall goal is to minimize the number of function evaluations to find high-performing permutations. The key challenge in solving this problem using the Bayesian optimization (BO) framework is to trade-off the complexity of statistical model and tractability of acquisition function optimization. In this paper, we propose and evaluate two algorithms for BO over Permutation Spaces (BOPS). First, BOPS-T employs Gaussian process (GP) surrogate model with Kendall kernels and a Tractable acquisition function optimization approach to select the sequence of permutations for evaluation. Second, BOPS-H employs GP surrogate model with Mallow kernels and a Heuristic search approach to optimize the acquisition function. We theoretically analyze the performance of BOPS-T to show that their regret grows sub-linearly. Our experiments on multiple synthetic and real-world benchmarks show that both BOPS-T and BOPS-H perform better than the state-of-the-art BO algorithm for combinatorial spaces. To drive future research on this important problem, we make new resources and real-world benchmarks available to the community.

Bayesian optimization (BO) is an efficient framework for solving black-box optimization problems with expensive function evaluations. This paper addresses the BO problem setting for combinatorial spaces (e.g., sequences and graphs) that occurs naturally in science and engineering applications. A prototypical example is molecular optimization guided by expensive experiments. The key challenge is to balance the complexity of statistical models and tractability of search to select combinatorial structures for evaluation. In this paper, we propose an efficient approach referred as Mercer Features for Combinatorial Bayesian Optimization (MerCBO). The key idea behind MerCBO is to provide explicit feature maps for diffusion kernels over discrete objects by exploiting the structure of their combinatorial graph representation. These Mercer features combined with Thompson sampling as the acquisition function allows us to employ efficient solvers for finding the next structure for evaluation. Experimental evaluation on diverse real-world benchmarks demonstrates that MerCBO performs similarly or better than prior methods.

Mobile system-on-chips (SoCs) are growing in their complexity and heterogeneity (e.g., Arm’s Big-Little architecture) to meet the needs of emerging applications, including games and artificial intelligence. This makes it very challenging to optimally manage the resources (e.g., controlling the number and frequency of different types of cores) at runtime to meet the desired trade-offs among multiple objectives such as performance and energy. This paper proposes a novel information-theoretic framework referred to as PaRMIS to create Pareto-optimal resource management policies for given target applications and design objectives. PaRMIS specifies parametric policies to manage resources and learns statistical models from candidate policy evaluation data in the form of target design objective values. The key idea is to select a candidate policy for evaluation in each iteration guided by statistical models that maximize the information gain about the true Pareto front. Experiments on a commercial heterogeneous SoC show that PaRMIS achieves better Pareto fronts and is easily usable to optimize complex objectives (e.g., performance per Watt) when compared to prior methods

Resistive random-access memory (ReRAM) is a promising technology for designing hardware accelerators for deep neural network (DNN) inferencing. However, stochastic noise in ReRAM crossbars can degrade the DNN inferencing accuracy. We propose the design and optimization of a high-performance, area-and energy-efficient ReRAM-based hardware accelerator to achieve robust DNN inferencing in the presence of stochastic noise. We make two key technical contributions. First, we propose a stochastic-noise-aware training method, referred to as ReSNA, to improve the accuracy of DNN inferencing on ReRAM crossbars with stochastic noise. Second, we propose an information-theoretic algorithm, referred to as CF-MESMO, to identify the Pareto set of solutions to trade-off multiple objectives, including inferencing accuracy, area overhead, execution time, and energy consumption. The main challenge in this context is that executing the ReSNA method to evaluate each candidate ReRAM design is prohibitive. To address this challenge, we utilize the continuous-fidelity evaluation of ReRAM designs associated with prohibitive high computation cost by varying the number of training epochs to trade-off accuracy and cost. CF-MESMO iteratively selects the candidate ReRAM design and fidelity pair that maximizes the information gained per unit computation cost about the optimal Pareto front. Our experiments on benchmark DNNs show that the proposed algorithms efficiently uncover high-quality Pareto fronts. On average, ReSNA achieves 2.57% inferencing accuracy improvement for ResNet20 on the CIFAR-10 dataset with respect to the baseline configuration. Moreover, CF-MESMO algorithm achieves 90.91% reduction in computation cost compared to the popular multi-objective optimization algorithm NSGA-II to reach the best solution from NSGA-II.

We consider the problem of optimizing hybrid structures (mixture of discrete and continuous input variables) via expensive black-box function evaluations. This problem arises in many real-world applications. For example, in materials design optimization via lab experiments, discrete and continuous variables correspond to the presence/absence of primitive elements and their relative concentrations respectively. The key challenge is to accurately model the complex interactions between discrete and continuous variables. In this paper, we propose a novel approach referred as Hybrid Bayesian Optimization (HyBO) by utilizing diffusion kernels, which are naturally defined over continuous and discrete variables. We develop a principled approach for constructing diffusion kernels over hybrid spaces by utilizing the additive kernel formulation, which allows additive interactions of all orders in a tractable manner. We theoretically analyze the modeling strength of additive hybrid kernels and prove that it has the universal approximation property. Our experiments on synthetic and six diverse real-world benchmarks show that HyBO significantly outperforms the state-of-the-art methods.

We consider the problem of black-box multi-objective optimization (MOO) using expensive function evaluations (also referred to as experiments), where the goal is to approximate the true Pareto set of solutions by minimizing the total resource cost of experiments. For example, in hardware design optimization, we need to find the designs that trade-off performance, energy, and area overhead using expensive computational simulations. The key challenge is to select the sequence of experiments to uncover high-quality solutions using minimal resources. In this paper, we propose a general framework for solving MOO problems based on the principle of output space entropy (OSE) search: select the experiment that maximizes the information gained per unit resource cost about the true Pareto front. We appropriately instantiate the principle of OSE search to derive efficient algorithms for the following four MOO problem settings: 1) The most basic single-fidelity setting, where experiments are expensive and accurate; 2) Handling black-box constraints which cannot be evaluated without performing experiments; 3) The discrete multi-fidelity setting, where experiments can vary in the amount of resources consumed and their evaluation accuracy; and 4) The continuous-fidelity setting, where continuous function approximations result in a huge space of experiments. Experiments on diverse synthetic and real-world benchmarks show that our OSE search based algorithms improve over state-of-the-art methods in terms of both computational-efficiency and accuracy of MOO solutions.

We study the novel problem of blackbox optimization of multiple objectives via multi-fidelity function evaluations that vary in the amount of resources consumed and their accuracy. The overall goal is to appromixate the true Pareto set of solutions by minimizing the resources consumed for function evaluations. For example, in power system design optimization, we need to find designs that trade-off cost, size, efficiency, and thermal tolerance using multi-fidelity simulators for design evaluations. In this paper, we propose a novel approach referred as Multi-Fidelity Output Space Entropy Search for Multi-objective Optimization (MF-OSEMO) to solve this problem. The key idea is to select the sequence of candidate input and fidelity-vector pairs that maximize the information gained about the true Pareto front per unit resource cost. Our experiments on several synthetic and real-world benchmark problems show that MF-OSEMO, with both approximations, significantly improves over the state-of-the-art single-fidelity algorithms for multi-objective optimization. Please note: A corrigendum was submitted for this paper on 24 September 2020.

We consider the problem of multi-objective (MO) blackbox optimization using expensive function evaluations, where the goal is to approximate the true Pareto set of solutions while minimizing the number of function evaluations. For example, in hardware design optimization, we need to find the designs that trade-off performance, energy, and area overhead using expensive simulations. We propose a novel uncertainty-aware search framework referred to as USeMO to efficiently select the sequence of inputs for evaluation to solve this problem. The selection method of USeMO consists of solving a cheap MO optimization problem via surrogate models of the true functions to identify the most promising candidates and picking the best candidate based on a measure of uncertainty. We also provide theoretical analysis to characterize the efficacy of our approach. Our experiments on several synthetic and six diverse real-world benchmark problems show that USeMO consistently outperforms the state-of-the-art algorithms.

Efficiency of power management system (PMS) is one of the key performance metrics for highly integrated system on chips (SoCs). Towards the goal of improving power efficiency of SoCs, we make two key technical contributions in this paper. First, we develop a multi-output switched-capacitor voltage regulator (SCVR) with a new flying capacitor crossing technique (FCCT) and cloud-capacitor method. Second, to optimize the design parameters of SCVR, we introduce a novel machine¬learning (ML)-inspired optimization framework to reduce the number of expensive design simulations. Simulation shows that power loss of the multi-output SCVR with FCCT is reduced by more than 40% compared to conventional multiple single-output SCVRs. Our ML-based design optimization framework is able to achieve more than 90% reduction in the number of simulations needed to uncover optimized circuit parameters of the proposed SCVR.

Autonomous electric mobility on demand (AEMoD) services are anticipated to be the future of private transportation, serving tens-of-thousands of requests per minute in large cities. To cope with this massive demand, a decentralized (i.e., zone-based) and multi-class management framework of AEMoD fleets was recently introduced. Yet, the inter-zone management of such approach has not been investigated. This paper thus fills this gap by studying the fleet re-balancing problem, with possible in-route charging, in decentralized multiclass AEMoD systems. A queuing model for multi-class re-balancing and possible in-route charging is developed on top of the system's decentralized fleet management. The stability conditions of this model are first derived, then the optimal inter-zone multi-class re-balancing and in-route charging decisions are derived so as to minimize the maximum response time in each deficient zone. Closed-form solutions are derived using Lagrangian analysis and simulations in a realistic setting in the city of Seattle are employed to illustrate the merits of our proposed re-balancing scheme as opposed to different baseline rebalancing approaches.

Many real-world edge applications including object detection, robotics, and smart health are enabled by deploying deep neural networks (DNNs) on energy-constrained mobile platforms. In this article, we propose a novel approach to trade off energy and accuracy of inference at runtime using a design space called Learning Energy Accuracy Tradeoff Networks (LEANets). The key idea behind LEANets is to design classifiers of increasing complexity using pretrained DNNs to perform input-specific adaptive inference. The accuracy and energy consumption of the adaptive inference scheme depends on a set of thresholds, one for each classifier. To determine the set of threshold vectors to achieve different energy and accuracy tradeoffs, we propose a novel multiobjective optimization approach. We can select the appropriate threshold vector at runtime based on the desired tradeoff. We perform experiments on multiple pretrained DNNs including ConvNet, VGG-16, and MobileNet using diverse image classification datasets. Our results show that we get up to a 50% gain in energy for negligible loss in accuracy, and optimized LEANets achieve significantly better energy and accuracy tradeoff when compared to a state-of-the-art method referred to as Slimmable neural networks.

Despite the significant advances in vehicle automation and electrification, the next-decade aspirations for massive deployments of autonomous electric mobility on demand (AEMoD) services in big cities are still threatened by two major bottlenecks, namely, the communication/computation and charging delays. In order to target the communication/computation delays, the paper suggests the exploitation of fog-based architectures for localized AEMoD system operations. These emerging architectures are soon to become widely used, allowing for all localized operational decisions to be made with very low latency by fog controllers located close to the end applications (e.g., each city zone for AEMoD systems). As for the charging delays, an optimized multi-class charging and dispatching queuing model, with partial charging option for AEMoD vehicles is developed for each of these zones. The stability conditions of this model and the optimal number of classes are then derived. The decisions on the proportions of each class vehicles to partially/fully charge or directly serve customers are optimized to minimize the maximum and average system response times using convex optimization and Lagrangian analysis. The results show the merits of our proposed model and optimized decision scheme compared to both the always-charge and the equal-split scheme. Furthermore, the comparison of the maximum and average response time minimization results shows a very low variance in performance, which suggests by using the linear programming solution for lower complexity.

We consider the problem of multi-objective (MO) blackbox optimization using expensive function evaluations, where the goal is to approximate the true Pareto-set of solutions by minimizing the number of function evaluations. For example, in hardware design optimization, we need to find the designs that trade-off performance, energy, and area overhead using expensive simulations. We propose a novel approach referred to as Max-value Entropy Search for Multi-objective Optimization (MESMO) to solve this problem. MESMO employs an output-space entropy based acquisition function to efficiently select the sequence of inputs for evaluation for quickly uncovering high-quality solutions. We also provide theoretical analysis to characterize the efficacy of MESMO. Our experiments on several synthetic and real-world benchmark problems show that MESMO consistently outperforms state-of-the-art algorithms.

Mobility-on-Demand (MoD) systems using electric vehicles (EVs) are expected to play a significantly increasing role with urban transportation systems in the near future, to both cope with the massive increases in urban population and reduce carbon emissions. One inconvenience in MoD-EV systems is the need for some customers to perform in-routing charging for almost-out-of-charge EVs. In this paper, we propose a routing scheme that aims to reduce this inconvenience by minimizing the relative excess time spent by MoD-EV systems customers for in-route charging compared to the on-road trip time. By modeling the routing problem between multiple MoD-EV stations with in-route charging as a multi-server queuing system, we formulate our objective as a stochastic convex optimization problem that minimizes the average overall trip time for all customers relatively to their actual trip time without in-route charging. Both single and multiple charging units per charging station are considered in this paper and modeled as M/M/1 and M/M/c queues, respectively. For both types of queues, the optimal routing proportions are derived analytically using the Lagrangian analysis and the Karush-Kuhn-Tucker conditions. Simulation results show the merits of our proposed solution in both cases as compared to the shortest time and the random routing decisions. Finally, the proposed method is tested on a real-world scenario, and the computation times are calculated for different settings.

On-Demand electric vehicle (EV) systems are expected to play a significantly increasing role in near future urban transportation systems, to cope with the massive increases in urban population and reduce global carbon emissions. One inconvenience in MoD-EV systems is the need of some customers to perform in-routing charging, which may cause delays in the trip time. Moreover, the customer choice of which station to charge at is an operational issue for the MoD-EV service operator due to the different pricing for the charging at different stations. Given a connected system linking these EVs and charging stations to the operator, we propose a routing scheme that aims to reduce these inconveniences for both the customers and the operator. By modeling the routing problem between multiple MoD-EV stations with in-route charging as a multi-server queuing system, we formulate the joint problem of minimizing the average overall trip time for all customers, relative to their actual trip time without in-route charging, and the average overall cost of charging as a dual-objective stochastic convex optimization problem. Optimal routing decisions are then derived analytically for any arbitrary weighting of the two problem objectives. Simulation results show the significant merits of our proposed solution as compared to shortest time and random routing decisions. They also illustrate the trade-offs between the delay experienced by the customer and the charging cost for the operator.

On-Demand electric vehicle (EV) systems are expected to have a significantly increasing role in the future of transportation systems in urban areas, to cope with the tremendous increases in urban population and decrease global carbon emissions. An inconvenience in Mobility-on-Demand Electric Vehicle (MoD-EV) systems is the need for some customers to charge EVs before reaching their destinations, which may cause delays in the trip time. Local and in-route charging options are available but the system operator needs to manage the charging assignments of the different EVs as charging all EVs locally may result in large delays. Given a connected system, we propose a routing strategy that aims to decrease these charging delays for the customers by sending the customers to different charging stations either at the pick-up location or nearby charging stations to minimize their average total trip time while respecting the charging constraints and avoid roads congestion. The problem is then formulated by modeling the routing between multiple MoD-EV stations as a multi-server queuing system with an objective of minimizing the expected overall trip duration for all customers, relative to their actual trip time without charging as a convex optimization problem. Optimal routing decisions and actual trip times are then derived analytically and simulation results show the significant gains of our proposed model as compared to shortest path and random routing schemes.

Mobility-On-Demand (MoD) systems using electric vehicles (EVs) are expected to play a significantly increasing role in near future urban transportation systems, to cope with the massive increases in urban population and reduce carbon emissions. One inconvenience in MoD- EV systems is the need of some customers to perform in-routing charging. In this paper, we propose a routing scheme that aims to reduce the inconvenience of in-route charging in MoD-EV systems. By modeling the routing problem between multiple MoD-EV stations with in-route charging as a multi-server queuing system, we formulate our objective as a stochastic convex optimization problem that minimizes the average overall trip time for all customers relative to their actual trip time without in-route charging. Optimal routing decisions that achieve this goal are then derived analytically. Simulation results show the significant merits of our proposed solution as compared to shortest and random routing decisions.

Despite the significant advances in vehicle automation and electrification, the next-decade aspirations for massive deployments of autonomous electric mobility on demand (AEMoD) services are still threatened by two major bottlenecks, namely the computational and charging delays. This paper proposes a solution for these two challenges by suggesting the use of fog computing for AEMoD systems, and developing an optimized multi-class charging and dispatching scheme for its vehicles. A queuing model representing the proposed multi-class charging and dispatching scheme is first introduced. The stability conditions of this model and the number of classes that fit the charging capabilities of any given city zone are then derived. Decisions on the proportions of each class vehicles to partially/fully charge, or directly serve customers are then optimized using a stochastic linear program that minimizes the maximum response time of the system. Results show the merits of our proposed model and optimized decision scheme compared to both the always-charge and the equal split schemes.