We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular, the practical benefit of cooldown is reflected in the bound due to the absence of logarithmic terms. Further, we show that this surprisingly close match between optimization theory and practice can be exploited for learning-rate tuning: we achieve noticeable improvements for training 124M and 210M Llama-type models by (i) extending the schedule for continued training with optimal learning-rate, and (ii) transferring the optimal learning-rate across schedules.

Training a modern machine learning architecture on a new task requires extensive learning-rate tuning, which comes at a high computational cost. Here we develop new Polyak-type adaptive learning rates that can be used on top of any momentum method, and require less tuning to perform well. We first develop MoMo, a **Mo**mentum **Mo**del based adaptive learning rate for SGD-M (stochastic gradient descent with momentum). MoMo uses momentum estimates of the batch losses and gradients sampled at each iteration to build a model of the loss function. Our model also makes use of any known lower bound of the loss function by using truncation, e.g. most losses are lower-bounded by zero. The models is then approximately minimized at each iteration to compute the next step. We show how MoMo can be used in combination with any momentum-based method, and showcase this by developing MoMo-Adam - which is Adam with our new model-based adaptive learning rate. We show that MoMo attains a $\mathcal{O}(1/\sqrt{K})$ convergence rate for convex problems with interpolation, needing knowledge of no problem-specific quantities other than the optimal value. Additionally, for losses with unknown lower bounds, we develop on-the-fly estimates of a lower bound, that are incorporated in our model. We demonstrate that MoMo and MoMo-Adam improve over SGD-M and Adam in terms of robustness to hyperparameter tuning for training image classifiers on MNIST, CIFAR, and Imagenet, for recommender systems on the Criteo dataset, for a transformer model on the translation task IWSLT14, and for a diffusion model.

A majority of scientific publications on wireless networking is (partly) based on network simulation in order to evaluate the presented concepts, algorithms and architectures. In research related to 4G and 5G cellular communication, the system-level simulation SimuLTE is a widely used model. Despite its popularity, no results regarding an experimental validation of this model are publicly available up to now. This paper presents results of a first step in this direction: A small-scale completely shielded cellular testbed based on OpenAirInterface is set up and packet delay and inter-arrival time are measured in several low and high load scenarios. Afterwards, the identical scenarios are modeled in SimuLTE. The comparison of simulation and measurement results indicate a good accuracy of the model for low-load scenarios. However, in particular in overload conditions involving more than one packet source the results differ significantly. Subsequently, several adaptations to the model are proposed which lead to a better agreement of simulation and measurement results in the investigated scenarios.