Projection predictive inference is a decision theoretic Bayesian approach that decouples model estimation from decision making. Given a reference model previously built including all variables present in the data, projection predictive inference projects its posterior onto a constrained space of a subset of variables. Variable selection is then performed by sequentially adding relevant variables until predictive performance is satisfactory. Previously, projection predictive inference has been demonstrated only for generalized linear models (GLMs) and Gaussian processes (GPs) where it showed superior performance to competing variable selection procedures. In this work, we extend projection predictive inference to support variable and structure selection for generalized linear multilevel models (GLMMs) and generalized additive multilevel models (GAMMs). Our simulative and real-world experiments demonstrate that our method can drastically reduce the model complexity required to reach reference predictive performance and achieve good frequency properties.

Current black-box variational inference (BBVI) methods require the user to make numerous design choices – such as the selection of variational objective and approximating family – yet there is little principled guidance on how to do so. We develop a conceptual framework and set of experimental tools to understand the effects of these choices, which we leverage to propose best practices for maximizing posterior approximation accuracy. Our approach is based on studying the pre-asymptotic tail behavior of the density ratios between the joint distribution and the variational approximation, then exploiting insights and tools from the importance sampling literature. Our framework and supporting experiments help to distinguish between the behavior of BBVI methods for approximating low-dimensional versus moderate-to-high-dimensional posteriors. In the latter case, we show that mass-covering variational objectives are difficult to optimize and do not improve accuracy, but flexible variational families can improve accuracy and the effectiveness of importance sampling – at the cost of additional optimization challenges. Therefore, for moderate-to-high-dimensional posteriors we recommend using the (mode-seeking) exclusive KL divergence since it is the easiest to optimize, and improving the variational family or using model parameter transformations to make the posterior and optimal variational approximation more similar. On the other hand, in low-dimensional settings, we show that heavy-tailed variational families and mass-covering divergences are effective and can increase the chances that the approximation can be improved by importance sampling.

We examine the accuracy of black box variational posterior approximations for parametric models in a probabilistic programming context. The performance of these approximations depends on (1) how well the variational family approximates the true posterior distribution, (2) the choice of divergence, and (3) the optimization of the variational objective. We show that even when the true variational family is used, high-dimensional posteriors can be very poorly approximated using common stochastic gradient descent (SGD) optimizers. Motivated by recent theory, we propose a simple and parallel way to improve SGD estimates for variational inference. The approach is theoretically motivated and comes with a diagnostic for convergence and a novel stopping rule, which is robust to noisy objective functions evaluations. We show empirically, the new workflow works well on a diverse set of models and datasets, or warns if the stochastic optimization fails or if the used variational distribution is not good.

Multitask Learning (MTL) aims to solve several related problems at the same time exploiting the similarities between them. In particular, Support Vector Machines (SVMs) can be used for MTL, providing a model that is potentially more flexible than a classical SVM trained over the data of all the tasks, and which can use more information than a set of independent models trained over each one of the tasks. Nevertheless, a major drawback of these SVMs is the large number of hyperparameters if no simplifying assumptions are made, which prevents from using standard selection methods as grid or random searches. In particular, both the common kernel and the kernels associated to each one of the tasks have to be selected. In this paper, we propose an approach to choose these kernels based on Gaussian Processes (GPs), whose Bayesian perspective allows one to deal naturally with several parameters. In particular, a GP is trained for each task, and the resultant kernel parameters are transferred to the SVM-based MTL model. Several experiment in real-world datasets show empirically the usefulness of this approach and the advantages of the GP-based kernel selection method.

In this work we transfer several recent acceleration techniques for the FISTA algorithm for composite convex optimization to the context of problems amenable to be solved by Block Coordinate Descent and, in particular, we show how these improvements can be used to speed up the training of the Sparse Group Lasso model. Experiments in several real life datasets illustrate how the proposed method can outperform the state of the art approach in terms of both the number of inner FISTA and outer block iterations required and, more importantly, the number of function and gradient evaluations needed to achieve a certain precision.