Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the ABC condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.

We provide the first convergence guarantee for black-box variational inference (BBVI) with the reparameterization gradient. While preliminary investigations worked on simplified versions of BBVI (e.g., bounded domain, bounded support, only optimizing for the scale, and such), our setup does not need any such algorithmic modifications. Our results hold for log-smooth posterior densities with and without strong log-concavity and the location-scale variational family. Notably, our analysis reveals that certain algorithm design choices commonly employed in practice, such as nonlinear parameterizations of the scale matrix, can result in suboptimal convergence rates. Fortunately, running BBVI with proximal stochastic gradient descent fixes these limitations and thus achieves the strongest known convergence guarantees. We evaluate this theoretical insight by comparing proximal SGD against other standard implementations of BBVI on large-scale Bayesian inference problems.

Minimizing the inclusive Kullback-Leibler (KL) divergence with stochastic gradient descent (SGD) is challenging since its gradient is defined as an integral over the posterior. Recently, multiple methods have been proposed to run SGD with biased gradient estimates obtained from a Markov chain. This paper provides the first non-asymptotic convergence analysis of these methods by establishing their mixing rate and gradient variance. To do this, we demonstrate that these methods—which we collectively refer to as Markov chain score ascent (MCSA) methods—can be cast as special cases of the Markov chain gradient descent framework. Furthermore, by leveraging this new understanding, we develop a novel MCSA scheme, parallel MCSA (pMCSA), that achieves a tighter bound on the gradient variance. We demonstrate that this improved theoretical result translates to superior empirical performance.

Supporting timely data services using fresh data in data-intensive real-time applications, such as e-commerce and transportation management is desirable but challenging, since the workload may vary dynamically. To control the data service delay to be below the specified threshold, we develop a predictive as well as reactive method for database admission control. The predictive method derives the workload bound for admission control in a predictive manner, making no statistical or queuing-theoretic assumptions about workloads. Also, our reactive scheme based on formal feedback control theory continuously adjusts the database load bound to support the delay threshold. By adapting the load bound in a proactive fashion, we attempt to avoid severe overload conditions and excessive delays before they occur. Also, the feedback control scheme enhances the timeliness by compensating for potential prediction errors due to dynamic workloads. Hence, the predictive and reactive methods complement each other, enhancing the robustness of real-time data services as a whole. We implement the integrated approach and several baselines in an open-source database. Compared to the tested open-loop, feedback-only, and statistical prediction + feedback baselines representing the state of the art, our integrated method significantly improves the average/transient delay and real-time data service throughput.

It is essential to process real-time data service requests such as stock quotes and trade transactions in a timely manner using fresh data, which represent the current real-world phenomena such as the stock market status. Users may simply leave when the database service delay is excessive. Also, temporally inconsistent data may give an outdated view of the real-world status. However, supporting the desired timeliness and freshness is challenging due to dynamic workloads. To address the problem, we present new approaches for 1) database backlog estimation, 2) fine-grained closed-loop admission control based on the backlog model, and 3) incoming load smoothing. Our backlog estimation and control-theoretic approaches aim to support the desired service delay bound without degrading the data freshness, critical for real-time data services. Specifically, we design, implement, and evaluate two feedback controllers based on linear control theory and fuzzy logic control theory, to meet the desired service delay. Workload smoothing, under overload, helps the database admit and process more transactions in a timely fashion by probabilistically reducing the burstiness of incoming data service requests. In terms of the data service delay and throughput, our closed-loop admission control and probabilistic load smoothing schemes considerably outperform several baselines in the experiments undertaken in a stock trading database testbed.

Real-time data services can benefit data-intensive real-time applications, e.g., e-commerce, via timely transaction processing using fresh data, e.g., the current stock prices. To enhance the real-time data service quality, we present several novel techniques for (1) database backlog estimation, (2) fine-grained closed-loop admission control based on the backlog model, and (3) hint-based incoming load smoothing. Our backlog estimation and feedback control aim to support the desired service delay bound without degrading the data freshness critical for real-time data services. Workload smoothing, under overload, help the database admit and process more transactions in a timely manner by probabilistically reducing the burstiness of incoming data service requests. In terms of the data service delay and throughput, our feedback-based admission control and probabilistic load smoothing considerably outperform the baselines, which represent the current state of the art, in the experiments performed in a stock trading database testbed.

It is challenging to manage the performance of real-time databases (RTDBs) that are often used in data-intensive real-time applications such as agile manufacturing and target tracking. Feedback control has recently been considered a promising approach to enabling reliable real-time data service. However, most existing work on feedback control of RTDB performance is not based on a RTDB-specific control model, which is critical for closed-loop system design. To address this problem, we design a novel RTDB model that can capture RTDB dynamics by modeling the relation between the total arrival rate (sum of the transaction arrival rate and restart rate) and utilization via a difference equation. Based on the model, we design and tune a utilization controller to compute the total arrival rate adjustment needed to support the desired average/transient utilization. We also design a QoS management scheme and admission control technique that can judiciously adapt the transaction QoS and arrival rate in a RTDB, if necessary, to support the desired utilization, while enhancing the success ratio. In a simulation study, we show that our approach can support the desired average/transient utilization for a range of transaction arrival rates under severe data contention, while considerably improving the success ratio compared to the tested baselines

A lot of real-time database (RTDB) research has been done to process transactions in a timely fashion using fresh data reflecting the current real world status. However, most existing RTDB work is based on simulations. Due to the absence of a publicly available RTDB testbed, it is very hard to evaluate real-time data management techniques in a realistic environment. To address the problem, we design and develop an initial version of a RTDB testbed, called Chronos, atop an open source database [5]. We develop soft real-time database workloads that model online stock trades, providing several knobs to specify workloads for RTDB performance evaluation. In addition, we develop a QoS management scheme in Chronos to detect overload and reduce workloads, via admission control and adaptive temporal data updates, under overload. From the extensive experiments using the stock trading workloads developed in Chronos, we observe that adaptive updates can considerably improve the transaction timeliness. We also observe that admission control can only enhance the timeliness under severe overload, possibly causing underutilization problems for moderate workloads. Overall, we observe several results different from simulation-based work, warranting the need for RTDB performance evaluation in a real database system.

It is challenging to process transactions in a timely fashion using fresh data, e.g., current stock prices, since database workloads may considerably vary due to dynamic data/resource contention. Further, transaction timeliness and data freshness requirements may compete for system resources. In this paper, we propose a novel feedback control model to support the desired data service delay by managing the size of the ready queue, which indicates the amount of the backlog in the database. We also propose a new self-adaptive update policy to adapt the freshness of cold data in a differentiated manner based on temporal data access and update patterns. Unlike most existing work on feedback control of real-time database (RTDB) performance, we actually implement and evaluate feedback control and database workload adaptation techniques in a real database testbed modeling stock trades. For performance evaluation, we undertake experiments in the testbed, which consists of thousands of client threads concurrently requesting database services for stock quotes, trades, and portfolio updates in a bursty manner. In these experiments, our database system supports the desired response time bound and data freshness, while processing a significantly larger number of transactions in time compared to the tested baselines.