Buyers (e.g., advertisers) often have limited financial or processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling, and any change may affect what buyers pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is an unbiased estimator when all the bidders in the same auction are randomly assigned to A or B but the observed treatment effect can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be much reduced if separate budgets are maintained for the A and B arms of the experiment.

Display ads proliferate on the web, but are they effective? Or are they irrelevant in light of all the other advertising that people see? We describe a way to answer these questions, quickly and accurately, without randomized experiments, surveys, focus groups or expert data analysts. Doubly robust estimation protects against the selection bias that is inherent in observational data, and a nonparametric test that is based on irrelevant outcomes provides further defense. Simulations based on realistic scenarios show that the resulting estimates are more robust to selection bias than traditional alternatives, such as regression modeling or propensity scoring. Moreover, computations are fast enough that all processing, from data retrieval through estimation, testing, validation and report generation, proceeds in an automated pipeline, without anyone needing to see the raw data.

Transaction data can arrive at a ferocious rate in the order that transactions are completed. The data contain an enormous amount of information about customers, not just transactions, but extracting up-to-date customer information from an ever changing stream of data and mining it in real-time is a challenge. This paper describes a statistically principled approach to designing short, accurate summaries or signatures of high dimensional customer behavior that can be kept current with a stream of transactions. A signature database can then be used for data mining and to provide approximate answers to many kinds of queries about current customers quickly and accurately, as an empirical study of the calling patterns of 96,000 wireless customers who made about 18 million wireless calls over a three month period shows.

Data-call records, internet packet headers, or other transaction records-are coming down a pipe at a ferocious rate, and we need to monitor statistics of the data.There is no reason to think that the data are normally distributed, so quantiles of the data are important to watch.The probe attached to the pipe has only limited memory, though, so it is impossible to compute the quantiles by sorting the data.The only possibility is to incrementally estimate the quantiles as the data fly by.This paper provides such an incremental quantile estimator.It resembles an exponentially weighted moving average in form, processing and memory requirements, but it is based on stochastic approximation so we call it an exponentially weighted stochastic approximation or EWSA.Simulations show that the EWSA outperforms other kinds of incremental estimates that also require minimal main memory, especially when extreme quantiles are tracked for patterns of behavior that change over time.Use of the EWSA is illustrated in an application to tracking call duration for a set of callers over a three month period.