Online consumer data presents new opportunities for measuring the effects of advertising. We combine search query data with the television commercial schedule for the 2011 Super Bowl to measure the causal impact of TV advertising on consumer search behavior. Examining 46 Super Bowl commercials, we generally find large spikes in search behavior related to the advertiser or product within 15 seconds following the conclusion of the TV commercial. We present results for four categories of Super Bowl advertisers: movies, online services, cars, and other consumer goods. We also consider the economic and statistical feasibility of scaling this analysis to use search data to measure the impact of other TV commercials.

We study the impact of display advertising on user search behavior using a field experiment. In such an experiment, the treatment group users are exposed to some display advertising campaign, while the control group users are not. During the campaign and the post-campaign period we monitor the user search queries and we label them as relevant or irrelevant to the campaign using techniques that leverage the bipartite query-URL click graph. Our results indicate that users who are exposed to the advertising campaign submit 5% to 25% more queries that are relevant to it compared to the unexposed users.

The department-store retailer John Wanamaker famously stated, "Half the money I spend on advertising is wasted--I just don't know which half." Compared with the measurement of advertising effectiveness in traditional media, online advertisers and publishers have considerable data advantages, including individual-level data on advertising exposures, clicks, searches, and other online user behaviors. However, as I shall discuss in this talk, the science of advertising effectiveness requires more than just quantity of data - even more important is the quality of the data. In particular, in many cases, using various statistical techniques with observational data leads to incorrect measurements. To measure the true causal effects, we run controlled experiments that suppress advertising to a control group, much like the placebo in a drug trial. With experiments to determine the ground truth, we can show that in many circumstances, observational-data techniques rely on identifying assumptions that prove to be incorrect, and they produce estimates differing wildly from the truth. Despite increases in data availability, Wanamaker's complaint remains just as true for online advertising as it was for print advertising a century ago.

Measuring the causal effects of online advertising (adfx) on user behavior is important to the health of the WWW publishing industry. In this paper, using three controlled experiments, we show that observational data frequently lead to incorrect estimates of adfx. The reason, which we label "activity bias," comes from the surprising amount of time-based correlation between the myriad activities that users undertake online. In Experiment 1, users who are exposed to an ad on a given day are much more likely to engage in brand-relevant search queries as compared to their recent history for reasons that had nothing do with the advertisement. In Experiment 2, we show that activity bias occurs for page views across diverse websites. In Experiment 3, we track account sign-ups at a competitor's (of the advertiser) website and find that many more people sign-up on the day they saw an advertisement than on other days, but that the true "competitive effect" was minimal. In all three experiments, exposure to a campaign signals doing "more of everything" in given period of time, making it difficult to find a suitable "matched control" using prior behavior. In such cases, the "match" is fundamentally different from the exposed group, and we show how and why observational methods lead to a massive overestimate of adfx in such circumstances.

North ads, or sponsored listings appearing just above the organic search results, generate the majority of clicks and revenues for search engines. In this paper, we ask whether the competing north ads exert externalities on each other. In particular, does increasing the number of rival north ads decrease the number of clicks I receive on my own north ad? We conduct a controlled experiment to investigate this question and find, to our surprise, that additional rival ads in the north tend to increase rather than decrease the click-through rate (CTR) of the top sponsored listing. We propose several possible explanations for this phenomenon, and point out directions for new theoretical models of sponsored search.