Bayesian optimization (BO) is a key technique for solving black-box optimization problems. This study extends the scope of BO from conventional applications (e.g., AutoML and robotics learning) to automated tuning of software systems. Despite GP (Gaussian Process) implementing a foundation formalism for exploitation and exploration in BO, its limited predictive power and unrealistic assumptions (e.g., continuity and Gaussianity) can severely affect its effectiveness and efficiency in tuning complex software systems. To overcome these limitations, we propose a BO framework CoffeeBoost, which implements exploitation and exploration with a GBDT-native distribution-free probabilistic surrogate model. CoffeeBoost constructs surrogate models via stochastic gradient boosting ensembles (SGBE) and quantifies probabilistic distributions via distribution-free conformal predictive systems. Moreover, CoffeeBoost leverages the residual paths in SGBE to improve the local adaptiveness of the resulting predictive distributions in a GBDT-native manner. Across eight auto-tuning benchmarks for database management systems (DBMS), we evaluate CoffeeBoost and show its superior learnability and optimizability against existing GP-based and tree-ensemble-based BO schemes. Detailed analysis further shows CoffeeBoost's predictive distributions excel in both coverage and tightness.

A fundamental problem in neural network theory is the quantification of uncertainty as it propagates through these constructs. Such quantification is crucial as neural networks become integrated into broader engineered systems that render decisions based on their outputs. In this paper, we engage the problem of estimating uncertainty in feedforward neural network constructs. Mathematically, the problem, in essence, amounts to understanding how the moments of an input distribution become modifies as they move through network layers. Despite its straightforward formulation, the nonlinear nature of modern feedforward architectures makes this is a mathematically challenging problem. Most contemporary approaches rely on some form of Monte Carlo sampling to construct inter-laminar distributions. Here, we borrow an approach from the control systems community known as quasilinear approximation, to enable a more analytical approach to the uncertainty quantification problem in this setting. Specifically, by using quasilinear approximation, nonlinearities are linearized in terms of the expectation of their gain in an input-output sense. We derive these expectations for several commonly used nonlinearities, under the assumption of Gaussian inputs. We then establish that the ensuing approximation is accurate relative to traditional linearization. Furthermore, we provide a rigorous example how this method can enable formal estimation of uncertainty in latent variables upstream of the network, within a target-tracking case study.

Gaussian Process (GP)-based Bayesian optimization (BO), i.e., GP-BO, emerges as a prevailing model-based framework for DBMS (Database Management System) auto-tuning. However, recent work shows GP-BO-based DBMS auto-tuners are significantly outperformed by auto-tuners based on SMAC, which features random forest surrogate models; such results motivate us to rethink and investigate the limitations of GP-BO in auto-tuner design. We find that the fundamental assumptions of GP-BO are widely violated when modeling and optimizing DBMS performance, while tree-ensemble-BOs (e.g., SMAC) can avoid the assumption pitfalls and deliver improved tuning efficiency and effectiveness. Moreover, we argue that existing tree-ensemble-BOs restrict further advancement in DBMS auto-tuning. First, existing tree-ensemble-BOs can only achieve distribution-free point estimates, but still impose unrealistic distributional assumptions on uncertainty (interval) estimates, which can compromise surrogate modeling and distort the acquisition function. Second, recent advances in (ensemble) gradient boosting, which can further enhance surrogate modeling against vanilla GP and random forest counterparts, have rarely been applied in optimizing DBMS auto-tuners. To address these issues, we propose a novel model-based DBMS auto-tuner, Centrum . Centrum achieves and improves distribution-free point and interval estimation in surrogate modeling with a two-phase learning procedure of stochastic gradient boosting ensembles (SGBE). Moreover, Centrum adopts a generalized SGBE-estimated locally-adaptive conformal prediction to facilitate a distribution-free interval (uncertainty) estimation and acquisition function. To our knowledge, Centrum is the first auto-tuner that realizes distribution-freeness to stress and enhance BO's practicality in DBMS auto-tuning, and the first to seamlessly fuse gradient boosting ensembles and conformal inference in BO. Extensive physical and simulation experiments on two DBMSs and three workloads show that Centrum outperforms 21 state-of-the-art (SOTA) DBMS auto-tuners based on BO with GP, random forest, gradient boosting, OOB (Out-Of-Bag) conformal ensemble and other surrogates, as well as that based on reinforcement learning and genetic algorithms.