LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples.We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples.Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility.For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations.Our method can be run using different compute time budgets.Under a limited budget, it outperforms prior automatic prompting methods on text simplification and mathematical reasoning (GSM8K, DeepMath, Math500), while achieving second-best results on classification and summarization and third-best on MedQA.With an extended, yet still modest budget, PIAST sets a new state of the art among automatic prompting methods on classification, simplification, GSM8K, DeepMath, and Math500.Overall, our results suggest that optimizing in-context examples, rather than exhaustively searching over instruction rewrites is the dominant lever for fast and data-efficient prompt engineering.Our code is available at

Proposed transformations primitivesSearch in the space of proposed transformationsFound transformation primitives and parameters Fig. 1: An overview of NSA: given training input-output pairs, a transformer proposes candidate transformation primitives, which the ARGA [6] symbolic search composes into a program whose resulting transformation is applied to the test input to produce the final prediction.

Many crucial problems in deep learning and statistical inference are caused by a variational gap, i.e., a difference between model evidence (log-likelihood) and evidence lower bound (ELBO). In particular, in a classical VAE setting that involves training via an ELBO cost function, it is difficult to provide a robust comparison of the effects of training between models, since we do not know a log-likelihood of data (but only its lower bound). In this paper, to deal with this problem, we introduce a general and effective upper bound, which allows us to efficiently approximate the evidence of data. We provide extensive theoretical and experimental studies of our approach, including its comparison to the other state-of-the-art upper bounds, as well as its application as a tool for the evaluation of models that were trained on various lower bounds.