The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC’s theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning’s complexity and the principle of parsimony.

We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal structure in transformer language models during training. By applying these refined LLCs (rLLCs) to individual components of a two-layer attention-only transformer, we gain novel insights into the progressive differentiation and specialization of attention heads. Our methodology reveals how attention heads differentiate into distinct functional roles over the course of training, analyzes the types of data these heads specialize to process, and discovers a previously unidentified multigram circuit. These findings demonstrate that rLLCs provide a principled, quantitative toolkit for developmental interpretability, which aims to understand models through their evolution across the learning process. This work advances the field of developmental interpretability by providing a mathematically rigorous approach to understanding neural networks through the lens of their learning process. More broadly, this work takes a step towards establishing the correspondence between data distributional structure, geometric properties of the loss landscape, learning dynamics, and emergent computational structures in neural networks.

We apply auction theory to develop a dynamic pricing framework for cloud computing sensitive to the priorities of the jobs being submitted. The framework automatically adjusts to unpredictable renewable power draws and changes in demand, leading to environmental and financial benefits.

Predictive control has received much attention from both theoreticians and practitioners in diverse fields. However, a challenging problem we must face is how to auto-tune the parameters of the predictive controller. So far, only a few guidelines related to tuning of the parameters of predictive controllers have been provided by the literature. In fact, these parameters are generally determined by the designer's experience. From the process control point of view, it is difficult to find out the optimal parameters for the control system based on the single quadratic performance index. In this paper, attempts are first made to introduce an existing basic predictive control strategy in order to arrive at a best tuning procedure. Later, several multi-objective optimization techniques are explored for the possibility of auto-tuning in model-based predictive control. Finally, the paper presents a predictive controller auto-timing procedure based on the developed recursive multi-objective optimization algorithm and simulate the properties of the scheme.

The paper presents a technique for performing dynamic goal-oriented buffer pool management for database management systems. To dynamically adjust the buffer pool sizes for the multiple buffer pools provided by database management systems is a complex constrained optimization problem. In the goal-oriented approach, the user specifies each buffer pool's random access response time goal and the total available number of buffers for all buffer pools. The problem is to dynamically expand or contract the buffer pool sizes based on the database workload to achieve these pre-defined response time goals for each buffer pool while maintaining the same total number of buffers in the database system. Our goal satisfaction algorithm monitors goal satisfaction of each buffer pool and periodically changes buffer pool sizes to improve goal satisfaction. The expansion and contraction process does not allocate new or free up existing virtual storage. We demonstrate that dynamic tuning can greatly improve buffer pool goal satisfaction through trace driven simulations.