We propose a novel asynchronous bundle method to solve distributed learning problems. Compared to existing asynchronous methods, our algorithm computes the next iterate based on a more accurate approximation of the objective function and does not require any prior information about the maximal information delay in the system. This makes the proposed method fast and easy to tune. We prove that the algorithm converges in both deterministic and stochastic (mini-batch) settings, and quantify how the convergence times depend on the level of asynchrony. The practical advantages of our method are illustrated through numerical experiments on classification problems of varying complexities and scales.

Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability compared to synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when and how decentralization can shorten the per-iteration runtimes. To support the decentralized training of transformer-based models, we introduce a decentralized Adam algorithm that overlaps communications with computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy our solution in clusters with up to 64 GPUs, demonstrating its practical advantages in both runtime and generalization performance under a fixed iteration budget. The experiment code is open-source at [https://github.com/WangZesen/Decentralized-Training-Exp](https://github.com/WangZesen/Decentralized-Training-Exp), and the extension code is open-source at [https://github.com/WangZesen/Decent-DP](https://github.com/WangZesen/Decent-DP).

This article studies the resilient leader-follower tracking problem for the frequency control of distributed energy sources(DESs), and focuses on scenarios with a high density of misbehaving agents. To tackle the problem, we propose a fully distributed resilient consensus protocol, which utilizes confidence weights to evaluate the level of trust among agents with a first-order filter and a softmax-type function. The protocol theoretically ensures that the system is uniformly ultimately bounded, even in the presence of high-density mis-behaving agents (that is, for each follower, it can have more misbehaving neighbors compared to normal neighbors). We also demonstrate the fast practical convergence of the proposed protocol with a simulation for a frequency control problem of DESs. Our study is a significant step towards enhancing the reliability and stability of modern power systems.

We propose a delay-agnostic asynchronous coordinate update algorithm (DEGAS) for computing operator fixed points, with applications to asynchronous optimization. DEGAS includes novel asynchronous variants of ADMM and block-coordinate descent as special cases. We prove that DEGAS converges with both bounded and unbounded delays under delay-free parameter conditions. We also validate by theory and experiments that DEGAS adapts well to the actual delays. The effectiveness of DEGAS is demonstrated by numerical experiments on classification problems.

In scalable machine learning systems, model training is often parallelized over multiple nodes that run without tight synchronization. Most analysis results for the related asynchronous algorithms use an upper bound on the information delays in the system to determine learning rates. Not only are such bounds hard to obtain in advance, but they also result in unnecessarily slow convergence. In this paper, we show that it is possible to use learning rates that depend on the actual time-varying delays in the system. We develop general convergence results for delay-adaptive asynchronous iterations and specialize these to proximal incremental gradient descent and block coordinate descent algorithms. For each of these methods, we demonstrate how delays can be measured on-line, present delay-adaptive step-size policies, and illustrate their theoretical and practical advantages over the state-of-the-art.

This paper presents a set of distributed second-order methods for addressing unconstrained, smooth convex optimization over fixed networks. The proposed second-order methods, referred to as Approximately-Zero-Gradient-Sum (AZGS) algorithms, allow each node to update by combining the Hessian inverse of its local objective and the estimates of its neighbors, so that the gradient sum of the local objectives can be sufficiently close to zero at each iteration. We show that the AZGS algorithms, with properly selected parameters, enable all the nodes to asymptotically reach a consensus that can be arbitrarily close to the optimal solution. Finally, simulation results demonstrate the effectiveness of the AZGS algorithms.