We study a nonconvex optimization algorithmic approach to phase retrieval and the more general problem of semidefinite low-rank matrix sensing. Specifically, we analyze the nonconvex landscape of a quartic Burer-Monteiro factored least-squares optimization problem. We develop a new analysis framework, taking advantage of the semidefinite problem structure, to understand the properties of second-order critical points—specifically, whether they (approximately) recover the ground truth matrix. We show that it can be helpful to (mildly) overparametrize the problem, that is, to optimize over matrices of higher rank than the ground truth. We then apply this framework to several well-studied problem instances: in addition to recovering existing state-of-the-art phase retrieval landscape guarantees (without overparametrization), we show that over-parametrizing by a factor at most logarithmic in the dimension allows recovery with optimal statistical sample complexity and error for the problems of (1) phase retrieval with sub-Gaussian measurements and (2) more general semidefinite matrix sensing with rank-1 Gaussian measurements. Previously, such statistical results had been shown only for estimators based on semidefinite programming. More generally, our analysis is partially based on the powerful method of convex dual certificates, suggesting that it could be applied to a much wider class of problems.

We introduce a new type of query mechanism for collecting human feedback, called the perceptual adjustment query (PAQ). Being both informative and cognitively lightweight, the PAQ adopts an inverted measurement scheme, and combines advantages from both cardinal and ordinal queries. We showcase the PAQ in the metric learning problem, where we collect PAQ measurements to learn an unknown Mahalanobis distance. This gives rise to a high-dimensional, low-rank matrix estimation problem to which standard matrix estimators cannot be applied. Consequently, we develop a two-stage estimator for metric learning from PAQs, and provide sample complexity guarantees for this estimator. We present numerical simulations demonstrating the performance of the estimator and its notable properties.

We present novel analysis and algorithms for solving sparse phase retrieval and sparse principal component analysis (PCA) with convex lifted matrix formulations. The key innovation is a new mixed atomic matrix norm that, when used as regularization, promotes low-rank matrices with sparse factors. We show that convex programs with this atomic norm as a regularizer provide near-optimal sample complexity and error rate guarantees for sparse phase retrieval and sparse PCA. While we do not know how to solve the convex programs exactly with an efficient algorithm, for the phase retrieval case we carefully analyze the program and its dual and thereby derive a practical heuristic algorithm. We show empirically that this practical algorithm performs similarly to existing state-of-the-art algorithms.

Overparametrized neural networks tend to perfectly fit noisy training data yet generalize well on test data. Inspired by this empirical observation, recent work has sought to understand this phenomenon of benign overfitting or harmless interpolation in the much simpler linear model. Previous theoretical work critically assumes that either the data features are statistically independent or the input data is high-dimensional; this precludes general nonparametric settings with structured feature maps. In this paper, we present a general and flexible framework for upper bounding regression and classification risk in a reproducing kernel Hilbert space. A key contribution is that our framework describes precise sufficient conditions on the data Gram matrix under which harmless interpolation occurs. Our results recover prior independent-features results (with a much simpler analysis), but they furthermore show that harmless interpolation can occur in more general settings such as features that are a bounded orthonormal system. Furthermore, our results show an asymptotic separation between classification and regression performance in a manner that was previously only shown for Gaussian features.

A common problem in recommendation systems is to learn a model of user preferences based only on comparisons of the relative attractiveness of different items. We consider this problem in the context of an ideal point model of user preference, where each user can be represented as a point in a low-dimensional space together with a set of items. In this model, the closer an item is to a user’s ideal point, the more that user prefers the item. When an embedding of items is known a priori, the problem of localizing a user’s ideal point from comparisons amongst items is well studied. However, relatively little work exists on learning embeddings for new items based only on such comparisons. In this paper, we consider the problem of embedding a set of items using paired comparisons from a set of known users. Specifically, we present a novel convex lifted method of learning the embedding representation p1,…,pn∈ Rnof n items given noisy responses of the form "user ukprefers item pito item pj" for an arbitrary set of users {uk} in Rd. We provide a range of simulations that validate the efficacy of our approach.

We consider the theory of regression on a manifold using reproducing kernel Hilbert space methods. Manifold models arise in a wide variety of modern machine learning problems, and our goal is to help understand the effectiveness of various implicit and explicit dimensionality-reduction methods that exploit manifold structure. Our first key contribution is to establish a novel nonasymptotic version of the Weyl law from differential geometry. From this we are able to show that certain spaces of smooth functions on a manifold are effectively finite-dimensional, with a complexity that scales according to the manifold dimension rather than any ambient data dimension. Finally, we show that given (potentially noisy) function values taken uniformly at random over a manifold, a kernel regression estimator (derived from the spectral decomposition of the manifold) yields minimax-optimal error bounds that are controlled by the effective dimension.