This position paper argues for two claims regarding AI testing and evaluation. First, to remain informative about deployment behaviour, evaluations need account for the possibility that AI systems understand their circumstances and reason strategically. Second, game-theoretic analysis can inform evaluation design by formalising and scrutinising the reasoning in evaluation-based safety cases. Drawing on examples from existing AI systems, a review of relevant research, and formal strategic analysis of a stylised evaluation scenario, we present evidence for these claims and motivate several research directions.

We study the robust regulation of incentive contracts. We consider the problem of a regulator choosing what contracts to authorize in the canonical principal-agent model with moral hazard. A firm (she) contracts with a worker (he), who then takes a costly productive action that yields a stochastic output. Hiring the worker imposes a fixed cost for the firm. The set of productive actions and the fixed cost define the firm's technology. The worker's actions being non-contractible, the firm incentivises production by offering an (authorised) contract that maps realised outputs to payments. We assume the firm and the worker are protected by limited liability, they are risk-neutral, and they maximise their profit and surplus. We add a third player to that standard model: the regulator (they). They choose the regulation, which is the set of authorised contracts the firm can offer. We assume that the regulator's payoff is a weighted sum of the firm's profit and the worker's surplus, where the latter has a (weakly) greater weight α ≥ 1. While the firm and the worker know the technology, the regulator has no information. They choose a regulation that minimises their worst-case regret.

Firms and platforms now routinely use online consumer data to offer customized prices, products, or advertising. This surge in data-driven market segmentation has sparked renewed academic and regulatory interest in the welfare implications of price discrimination. As demonstrated by [Bergemann et al., 2015], market segmentations can lead to a wide range of welfare outcomes and, in particular, be designed so as to maximize total consumer surplus. Yet, despite policymakers' concerns about the potential adverse effects of market segmentation on poorer consumers [The White House Council of Economic Advisers, 2015], little theoretical progress has been made regarding the heterogeneous welfare effects that market segmentation might have across consumers. This concern is all the more relevant given that segmentations that maximize consumers' surplus tend to primarily benefit consumers with a high willingness to pay, who are more likely to be richer [Condorelli, 2013, Dworczak et al., 2021].