New large language models (LLMs) are being released every day.Some perform significantly better or worse than expected given their parameter count.Therefore, there is a need for a method to independently evaluate models.The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests-an expensive operation since humans are ideally required to compare LLM outputs.We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score.This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge.

Floating-point numbers represent only a subset of real numbers. As such, floating-point arithmetic introduces approximations that can compound and have a significant impact on numerical simulations. We introduce encapsulated error, a new way to estimate the numerical error of an application and provide a reference implementation, the Shaman library. Our method uses dedicated arithmetic over a type that encapsulates both the result the user would have had with the original computation and an approximation of its numerical error. We thus can measure the number of significant digits of any result or intermediate result in a simulation. We show that this approach, although simple, gives results competitive with state-of-the-art methods. It has a smaller overhead, and it is compatible with parallelism, making it suitable for the study of large-scale applications.

Extensive work has been done to evaluate the numerical accuracy of computations. However, getting fine-grained information on the operations that caused the inaccuracies observed in a given output is still a hard problem. We propose a new method, under the name tagged error, to get fine information on the impact of user-defined code sections on the numerical error of any floating-point number in a program. Our method uses a dedicated arithmetic over a type that encapsulates both the result the user would have had with the original computation and an approximation of its numerical error stored as an unevaluated sum of terms that can each be attributed to a single source. It lets us quantify the impact of potential error sources on any output of a computation while taking phenomena such as error amplification or dampening, due to later operations, into account. Furthermore, we can use this information to do targeted modifications of an algorithm, improving both its speed and precision, as illustrated by a study on the conjugate gradient algorithm.