We consider statistical tasks in high dimensions whose loss depends on the data only through its projection into a fixed-dimensional subspace spanned by the parameter vectors and certain ground truth vectors. This includes classifying mixture distributions with cross-entropy loss with one and two-layer networks, and learning single and multi-index models with one and two-layer networks. When the data is drawn from an isotropic Gaussian mixture distribution, it is known that the evolution of a finite family of summary statistics under stochastic gradient descent converges to an autonomous ordinary differential equation (ODE), as the dimension and sample size go to $\infty$ and the step size goes to $0$ commensurately. Our main result is that these ODE limits are universal in that this limit is the same whenever the data is drawn from mixtures of arbitrary product distributions whose first two moments match the corresponding Gaussian distribution, provided the initialization and ground truth vectors are coordinate-delocalized. We complement this by proving two corresponding non-universality results. We provide a simple example where the ODE limits are non-universal if the initialization is coordinate aligned. We also show that the stochastic differential equation limits arising as fluctuations of the summary statistics around their ODE’s fixed points are not universal.

Self-correcting quantum memories store logical quantum information for exponential time in thermal equilibrium at low temperatures. By definition, these systems are slow mixing. This raises the question of how the memory state, which we refer to as the Gibbs state within a logical sector, is created in the first place.

We analyze the general biased adjacent transposition shuffle process, which is a well-studied Markov chain on the symmetric group Sn. In each step, an adjacent pair of elements i and j are chosen, and then i is placed ahead of j with probability pij. This Markov chain arises in the study of self-organizing lists in theoretical computer science, and has close connections to exclusion processes from statistical physics and probability theory. It is conjectured (see Fill (2003)) that for general pij satisfying pij ≥ 1/2 for all i 0, as long as pij >1/2+ε for all i

A common obstruction to efficient sampling from high-dimensional distributions with Markov chains is the multimodality of the target distribution because they may get trapped far from stationarity. Still, one hopes that this is only a barrier to the mixing of Markov chains from worst-case initializations and can be overcome by choosing high-entropy initializations, e.g., a product or weakly correlated distribution. Ideally, from such initializations, the dynamics would escape from the saddle points separating modes quickly and spread its mass between the dominant modes with the correct probabilities.

We rigorously study the joint evolution of training dynamics via stochastic gradient descent (SGD) and the spectra of empirical Hessian and gradient matrices. We prove that in two canonical classification tasks for multi-class high-dimensional mixtures and either 1 or 2-layer neural networks, the SGD trajectory rapidly aligns with emerging low-rank outlier eigenspaces of the Hessian and gradient matrices. Moreover, in multi-layer settings this alignment occurs per layer, with the final layer's outlier eigenspace evolving over the course of training, and exhibiting rank deficiency when the SGD converges to sub-optimal classifiers. This establishes some of the rich predictions that have arisen from extensive numerical studies in the last decade about the spectra of Hessian and information matrices over the course of training in overparametrized networks.

Sampling from the q-state ferromagnetic Potts model is a fundamental question in statistical physics, probability theory, and theoretical computer science. On general graphs, this problem is computationally hard, and this hardness holds at arbitrarily low temperatures. At the same time, in recent years, there has been significant progress showing the existence of low-temperature sampling algorithms in various specific families of graphs. Our aim in this paper is to understand the minimal structural properties of general graphs that enable polynomial-time sampling from the q-state ferromagnetic Potts model at low temperatures. We study this problem from the perspective of the widely-used Swendsen-Wang dynamics and the closely related random-cluster dynamics. These are non-local Markov chains that have long been believed to converge rapidly to equilibrium at low temperatures in many graphs. However, the hardness of the sampling problem likely indicates that this is not even the case for all bounded degree graphs. Our results demonstrate that a key graph property behind fast or slow convergence time for these dynamics is whether the independent edge-percolation on the graph admits a strongly supercritical phase. By this, we mean that at large $p\lt 1$, it has a large linear-sized component, and the graph complement of that component is comprised of only small components Specifically, we prove that such a condition implies fast mixing of the Swendsen-Wang and random-cluster dynamics on two general families of bounded-degree graphs: (a) graphs of at most stretched-exponential volume growth and (b) locally treelike graphs. In the other direction, we show that, even among graphs in those families, these Markov chains can converge exponentially slowly at arbitrarily low temperatures if the edge-percolation condition does not hold. In the process, we develop new tools for the analysis of non-local Markov chains, including a framework to bound the speed of disagreement propagation in the presence of long-range correlations, an understanding of spatial mixing properties on trees with random boundary conditions, and an analysis of burn-in phases at low temperatures.

An important paradigm in the understanding of mixing times of Glauber dynamics for spin systems is the correspondence between spatial mixing properties of the models and bounds on the mixing time of the dynamics. This includes, in particular, the classical notions of weak and strong spatial mixing, which have been used to show the best known mixing time bounds in the high-temperature regime for the Glauber dynamics for the Ising and Potts models. Glauber dynamics for the random-cluster model does not naturally fit into this spin systems framework because its transition rules are not local. In this paper, we present various implications between weak spatial mixing, strong spatial mixing, and the newer notion of spatial mixing within a phase, and mixing time bounds for the random-cluster dynamics in finite subsets of ℤd for general d  2. These imply a host of new results, including optimal O(N log N) mixing for the random cluster dynamics on torii and boxes on N vertices in ℤd at all high temperatures and at sufficiently low temperatures, and for large values of q quasi-polynomial (or quasi-linear when d = 2) mixing time bounds from random phase initializations on torii at the critical point (where by contrast the mixing time from worst-case initializations is exponentially large). In the same parameter regimes, these results translate to fast sampling algorithms for the Potts model on ℤd for general d. * The full version of the paper can be accessed at https://arxiv.org/abs/2207.11195

We consider the problem of sampling from the ferromagnetic Potts and random-cluster models on a general family of random graphs via the Glauber dynamics for the random-cluster model. The random-cluster model is parametrized by an edge probability p ∈ (0,1) and a cluster weight q > 0. We establish that for every q ≥ 1, the random-cluster Glauber dynamics mixes in optimal Θ(nlog n) steps on n-vertex random graphs having a prescribed degree sequence with bounded average branching γ throughout the full high-temperature uniqueness regime p < p_u(q,γ). The family of random graph models we consider includes the Erdős-Rényi random graph G(n,γ/n), and so we provide the first polynomial-time sampling algorithm for the ferromagnetic Potts model on Erdős-Rényi random graphs for the full tree uniqueness regime. We accompany our results with mixing time lower bounds (exponential in the largest degree) for the Potts Glauber dynamics, in the same settings where our Θ(n log n) bounds for the random-cluster Glauber dynamics apply. This reveals a novel and significant computational advantage of random-cluster based algorithms for sampling from the Potts model at high temperatures.

We study the scaling limits of stochastic gradient descent (SGD) with constant step-size in the high-dimensional regime. We prove limit theorems for the trajectories of summary statistics (i.e., finite-dimensional functions) of SGD as the dimension goes to infinity. Our approach allows one to choose the summary statistics that are tracked, the initialization, and the step-size. It yields both ballistic (ODE) and diffusive (SDE) limits, with the limit depending dramatically on the former choices. We find a critical scaling regime for the step-size below which this ``effective dynamics" matches gradient flow for the population loss, but at which, a new correction term appears which changes the phase diagram. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate. We demonstrate our approach on popular examples including estimation for spiked matrix and tensor models and classification via two-layer networks for binary and XOR-type Gaussian mixture models. These examples exhibit surprising phenomena including multimodal timescales to convergence as well as convergence to sub-optimal solutions with probability bounded away from zero from random (e.g., Gaussian) initializations.

Glauber dynamics on spin systems are well known to suffer exponential slowdowns at low temperatures due to the emergence of multiple metastable phases, separated by narrow bottlenecks that are hard for the dynamics to cross. It is a folklore belief that if the dynamics is initialized from an appropriate random mixture of ground states, one for each phase, then convergence to the Gibbs distribution should be much faster. However, such phenomena have largely evaded rigorous analysis, as most tools in the study of Markov chain mixing times are tailored to worst-case initializations.

Stochastic gradient descent (SGD) is a popular algorithm for optimization problems arising in high-dimensional inference tasks. Here one produces an estimator of an unknown parameter from independent samples of data by iteratively optimizing a loss function. This loss function is random and often non-convex. We study the performance of the simplest version of SGD, namely online SGD, from a random start in the setting where the parameter space is high-dimensional. We develop nearly sharp thresholds for the number of samples needed for consistent estimation as one varies the dimension. Our thresholds depend only on an intrinsic property of the population loss which we call the information exponent. In particular, our results do not assume uniform control on the loss itself, such as convexity or uniform derivative bounds. The thresholds we obtain are polynomial in the dimension and the precise exponent depends explicitly on the information exponent. As a consequence of our results, we find that except for the simplest tasks, almost all of the data is used simply in the initial search phase to obtain non-trivial correlation with the ground truth. Upon attaining non-trivial correlation, the descent is rapid and exhibits law of large numbers type behavior. We illustrate our approach by applying it to a wide set of inference tasks such as phase retrieval, and parameter estimation for generalized linear models, online PCA, and spiked tensor models, as well as to supervised learning for single-layer networks with general activation functions.

The random-cluster (FK) model is a key tool for the study of phase transitions and for the design of efficient Markov chain Monte Carlo (MCMC) sampling algorithms for the Ising/Potts model. It is well-known that in the high-temperature region beta 1 and p != p_c(q) the mixing time of the FK-dynamics is polynomial in n for every realizable boundary condition. Previously, for boundary conditions that do not carry long-range information (namely wired and free), Blanca and Sinclair (2017) had proved that the FK-dynamics in the same setting mixes in optimal O(n^2 log n) time. To illustrate the difficulties introduced by general boundary conditions, we also construct a class of non-realizable boundary conditions that induce slow (stretched-exponential) convergence at high temperatures.

La dynamique de Swendsen–Wang a été proposée à la fin des années 1980 comme une alternative à la dynamique du bain-de-chaleur à un site, dans laquelle des mises à jour globales permettent à cet algorithme MCMC de passer plus vite d’un état métastable à un état de mélange idéal. Gore et Jerrum (J. Stat. Phys. 97 (1999) 67–86) ont trouvé que cette dynamique peut en fait montrer un mélange lent: ils ont montré, pour le modèle de Potts à $q\geq 3$ couleurs sur le graphe complet sur $n$ sommets au point critique $\beta_{c}(q)$, que la dynamique de Swendsen–Wang vérifie $t_{\mathrm{mix}}\geq \exp(c\sqrt{n})$. Galanis et al. (In Proc. of the 19th International Workshop on Randomization and Computation (RANDOM 2015) (2015) 815–828) a montré que $t_{\mathrm{mix}}\geq \exp(cn^{1/3})$ dans toute la fenêtre critique $(\beta_{s},\beta_{S})$ autour de $\beta_{c}$, et Blanca et Sinclair (In Proc. of the 19th International Workshop on Randomization and Computation (RANDOM 2015) (2015) 528–543) ont établit que $t_{\mathrm{mix}}\geq \exp(c\sqrt{n})$ dans la fenêtre critique pour le modèle de champs moyen FK, ce qui implique la même borne pour Swendsen–Wang grâce des estimées de comparaison connues. Dans les deux cas, une borne supérieure de $t_{\mathrm{mix}}\leq \exp(c'n)$ était connue. Dans cet article, nous montrons que le temps de mélange est vraiment exponentiel en $n$: plus précisément, $t_{\mathrm{mix}}\geq \exp (cn)$ pour la dynamique de Swendsen–Wang quand $q\geq 3$ et $\beta\in(\beta_{s},\beta_{S})$, et la même borne est vraie pour l’algorithme MCMC associé pour le modèle de champs moyen FK quand $q>2$.