Model Cascading, recently applied successfully to LLMs, is a simple but powerful technique that improves the efficiency of inference by selectively applying models of varying sizes. Models are used in sequence from smallest to largest, only deferring samples to large, costly models when smaller models are not sufficiently confident. Existing approaches to deferral use only limited small model confidence estimates because of the inaccessibility of the large model, although large model confidence is known to be important. We therefore propose a bi-directional approach to deferral that considers the confidence of small and large models in the cascade simultaneously through the use of a proxy for the large model. This requires a richer representation of model confidence to enable comparative calibration: we use an analysis of hidden states to improve post-invocation confidence of the small model, which in itself improves cascading results over prior approaches. We then combine this with a tiny proxy model to estimate pre-invocation confidence of the large model. We examine the proposed cascading system over challenging, multiple-choice datasets, finding improvements over standard cascading baselines reflected in reductions in deferrals to more costly models.

An important class of negotiation games that use human language do not have predefined ‘moves’: it is up to the agents in the game to define moves via natural language that will lead them towards their goal. In the context of other games, however, a notion of intents — structured moves from a predefined set — have been found to be useful. In this paper, we show that it is possible to define and learn post-hoc intents in a practical way for AI agents in a negotiation game, using a text-to-text Transformer model; we show that this improves agent performance, and further allows the definition of a wider range of agents for training.

The platform approach to hardware design was originally suggested as a method of broadening the domain of applications a custom VLSI chip can be applied to by introducing additional programmability. In this paper we extend the idea of platform design with the objective broadening the range of applications of FPGAs by making runtime resource allocation available for FPGAs that do not support run time reconfiguration natively. We explore this approach by constructing a virtual platform FPGA; that is a reconfigurable platform implemented on top of a native FPGA. The paper gives an overview of the application programmer's interface of our proposed virtual platform and some of the communication primitives that we have implemented to construct the virtual platform. We note that there is a need for more work in identifying computation primitives for a virtual platform that strike a balance between supporting useful abstractions and having efficient implementations on typical native FPGAs.