We study the problem of transfer learning, observing that previous efforts to understand its information-theoretic limits do not fully exploit the geometric structure of the source and target domains. In contrast, our study first illustrates the benefits of incorporating a natural geometric structure within a linear regression model, which corresponds to the generalized eigenvalue problem formed by the Gram matrices of both domains. We next establish a finite-sample minimax lower bound, propose a refined model interpolation estimator that enjoys a matching upper bound, and then extend our framework to multiple source domains and generalized linear models. Surprisingly, as long as information is available on the distance between the source and target parameters, negative-transfer does not occur. Simulation studies show that our proposed interpolation estimator outperforms state-of-the-art transfer learning methods in both moderate- and high-dimensional settings.

Distributionally robust learning (DRL) is increasingly seen as a viable method to train machine learning models for improved model generalization. These min-max formulations, however, are more difficult to solve. We provide a new stochastic gradient descent algorithm to efficiently solve this DRL formulation. Our approach applies gradient descent to the outer minimization formulation and estimates the gradient of the inner maximization based on a sample average approximation. The latter uses a subset of the data sampled without replacement in each iteration, progressively increasing the subset size to ensure convergence. We rigorously establish convergence to a near-optimal solution under standard regularity assumptions and, for strongly convex losses, match the best known $O(\epsilon{ −1})$ rate of convergence up to a known threshold. Empirical results demonstrate the significant benefits of our approach over previous work in improving learning for model generalization.

We consider the problem of estimating the Wasserstein distance between the empirical measure and a set of probability measures whose expectations over a class of functions (hypothesis class) are constrained. If this class is sufficiently rich to characterize a particular distribution (e.g., all Lipschitz functions), then our formulation recovers the Wasserstein distance to such a distribution. We establish a strong duality result that generalizes the celebrated Kantorovich-Rubinstein duality. We also show that our formulation can be used to beat the curse of dimensionality, which is well known to affect the rates of statistical convergence of the empirical Wasserstein distance. In particular, examples of infinite-dimensional hypothesis classes are presented, informed by a complex correlation structure, for which it is shown that the empirical Wasserstein distance to such classes converges to zero at the standard parametric rate. Our formulation provides insights that help clarify why, despite the curse of dimensionality, the Wasserstein distance enjoys favorable empirical performance across a wide range of statistical applications.

Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays in waiting for the slowest learners (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness that can adversely affect convergence. In this work we present the first theoretical characterization of the speed-up offered by asynchronous methods by analyzing the trade-off between the error in the trained model and the actual training runtime (wallclock time). The novelty in our work is that our runtime analysis considers random straggler delays, which helps us design and compare distributed SGD algorithms that strike a balance between stragglers and staleness. We also present a new convergence analysis of asynchronous SGD variants without bounded or exponential delay assumptions, and a novel learning rate schedule to compensate for gradient staleness.

We propose C-NORTA, an exact algorithm to generate random variates from the tail of a bivariate NORTA random vector. (A NORTA random vector is specified by a pair of marginals and a rank or product–moment correlation, and it is sampled using the popular NORmal-To-Anything procedure.) We first demonstrate that a rejection-based adaptation of NORTA on such constrained random vector generation problems may often be fundamentally intractable. We then develop the C-NORTA algorithm, relying on strategic conditioning of the NORTA vector, followed by efficient approximation and acceptance/rejection steps. We show that, in a certain precise asymptotic sense, the sampling efficiency of C-NORTA is exponentially larger than what is achievable through a naïve application of NORTA. Furthermore, for at least a certain class of problems, we show that the acceptance probability within C-NORTA decays only linearly with respect to a defined rarity parameter. The corresponding decay rate achievable through a naïve adaptation of NORTA is exponential. We provide directives for efficient implementation.

A basic requirement for running multiattribute auctions is knowledge of the utility function of the buyer that trades off nonprice attributes against price. We present and study an approach that elicits this preference structure based on a markovian polyhedral sampling scheme called the Hit-and-Run algorithm. An advantage of this technique is its relative simplicity - it relies only on matrix algebra as opposed to the use of nonlinear optimization techniques by other methods in the literature. Computational results suggest that this method is fast and accurate.