We develop a new class of self-tuning algorithms to solve a root-finding problem involving a Lipschitz continuous operator, with applications in convex optimization, minimax saddle point problems and variational inequalities. Our methods are adaptive to the unknown, problem specific parameters, such as the Lipschitz constant and the variance of the stochastic operator. Unlike prior work, our approach does not rely on restrictive assumptions, such as a bounded domain, boundedness of the operator or a light-tailed distribution. We prove a $\tilde{\mathcal{O}}(N^{-1/2})$ average-iterate convergence rate of the restricted merit function under an affine noise assumption, matching the optimal rate up to log factors. In addition, we improve the convergence rate to $\mathcal{O}(N^{-1})$ under a strong growth condition, characterizing the field of cutting-edge machine learning models and matching the optimal rate for the \textit{deterministic regime}. Finally, we illustrate the effectiveness of the proposed algorithms through numerical experiments on saddle point problems. Our results suggest that the adaptive step sizes automatically take advantage of the structure of the noise and observe improved convergence in certain settings, such as when the strong growth condition holds. To the best of our knowledge, this is the first method for root-finding problems under mild assumptions that adapts to the structure of the noise to obtain an improved convergence rate.

One underlying assumption of recent Federated Learning (FL) paradigms is that all local models share an identical network architecture. However, this assumption is inefficient for heterogeneous systems where devices possess varying computation and communication capabilities. The presence of such heterogeneity among devices negatively impacts the scalability of FL and slows down the training process due to the existence of stragglers. To this end, this paper proposes a novel federated compression framework for heterogeneous models , named FedHM, distributing the heterogeneous low-rank models to clients and then aggregating them into a full-rank global model. Furthermore, FedHM significantly reduces communication costs by utilizing low-rank models. Compared with state-of-the-art heterogeneous FL methods under various FL settings, FedHM is superior in the performance and robustness of models with different sizes. Additionally, the convergence guarantee of FL for heterogeneous devices is first theoretically analyzed.

We describe a line-search algorithm which achieves the best-known worst-case complexity results for problems with a certain “strict saddle” property that has been observed to hold in low-rank matrix optimization problems. Our algorithm is adaptive, in the sense that it makes use of backtracking line searches and does not require prior knowledge of the parameters that define the strict saddle property.