New large language models (LLMs) are being released every day.Some perform significantly better or worse than expected given their parameter count.Therefore, there is a need for a method to independently evaluate models.The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests-an expensive operation since humans are ideally required to compare LLM outputs.We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score.This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge.

We report the design and pilot testing of a digital application to help middle school children learn integers, Touchy-Pinchy Integers (TPI). The design rationale for this touchscreen-based learning system was based on a neutralization model of integers. To design this embodied mathematics learning system, we associated commonly used touchscreen interaction gestures with mathematical meaning related to integers. Users could manipulate two collinear towers of stacked positive and negative blocks, which together represent an integer expression. The learner could add either type of block, by tapping on the appropriate part of the screen. They could also subtract a desired number of either type of block, by slicing across the blocks. They could also neutralize a pair of positive and negative blocks, by pinching in. The pinch-out actions generated a new pair of positive and negative blocks. A flipping gesture reversed the polarity of the integer expression, which corresponded to a multiplication by -1. Results from a pilot study with grade 7s students indicated that the application helped students resolve deep conceptual issues related to negative numbers, through embodied learning. We present some of our observations from the study and interpret these episodes using embodied learning theory.

Despite advances in software modularity, security, and reliability,offline patching remains the predominant form of updating or protecting commodity software. Unfortunately, the mechanics of hot patching (the process of upgrading a program while it executes) remain understudied, even though such a capability offers practical benefits for both consumer and mission-critical systems. A reliable hot patching procedure would serve particularly well by reducing the downtime necessary for critical functionality or security upgrades. Yet, hot patching also carries the risk -- real or perceived -- of leaving the system in an inconsistent state, which leads many owners to forego its benefits as too risky. In this paper, we propose a novel method for hot patching ELF binaries that supports (a) synchronized global data and code updates and (b)reasoning about the results of applying the hot patch. We propose a format, which we call a Patch Object, for encoding patches as a special type of ELF relocatable object file. Our tool, Katana, automatically creates these patch objects as a by-product of the standard source build process. Katana also allows an end-user to apply the Patch Objects to a running process. In essence, our method can be viewed as an extension of the Application Binary Interface (ABI), and we argue for its inclusion in future ABI standards.

We present Darwin, an enabling technology for mobile phone sensing that combines collaborative sensing and classification techniques to reason about human behavior and context on mobile phones. Darwin advances mobile phone sensing through the deployment of efficient but sophisticated machine learning techniques specifically designed to run directly on sensor-enabled mobile phones (i.e., smartphones). Darwin tackles three key sensing and inference challenges that are barriers to mass-scale adoption of mobile phone sensing applications: (i) the human-burden of training classifiers, (ii) the ability to perform reliably in different environments (e.g., indoor, outdoor) and (iii) the ability to scale to a large number of phones without jeopardizing the "phone experience" (e.g., usability and battery lifetime). Darwin is a collaborative reasoning framework built on three concepts: classifier/model evolution, model pooling, and collaborative inference. To the best of our knowledge Darwin is the first system that applies distributed machine learning techniques and collaborative inference concepts to mobile phones. We implement the Darwin system on the Nokia N97 and Apple iPhone. While Darwin represents a general framework applicable to a wide variety of emerging mobile sensing applications, we implement a speaker recognition application and an augmented reality application to evaluate the benefits of Darwin. We show experimental results from eight individuals carrying Nokia N97s and demonstrate that Darwin improves the reliability and scalability of the proof-of-concept speaker recognition application without additional burden to users.

Virtualization has seen a rebirth for a wide variety of uses; in our field, systems security researchers routinely use it as a standard tool for providing isolation and introspection. Researchers' use of virtual machines has reached a level of orthodoxy that makes it difficult for the collective wisdom to consider alternative approaches to protecting computation. We suggest that many scenarios exist where virtual machines do not provide a suitable tool or appropriate security properties. We analyze the use of virtual machines in the systems security space and we highlight other work that questions the current (ab)uses of virtualization.