Traditional online quality metrics are based on search and browsing signals, such as position and time of the click. Such metrics typically model all users' behavior in exactly the same manner. Modeling individuals' behavior in Web search may be challenging as the user's historical behavior may not always be available (e.g., if the user is not signed into a given service). However, in personal search, individual users issue queries over their personal corpus (e.g. emails, files, etc.) while they are logged into the service. This brings an opportunity to calibrate online quality metrics with respect to an individual's search habits. With this goal in mind, the current paper focuses on a user-centric evaluation framework for personal search by taking into account variability of search and browsing behavior across individuals. The main idea is to calibrate each interaction of a user with respect to their historical behavior and search habits. To formalize this, a characterization of online metrics is proposed according to the relevance signal of interest and how the signal contributes to the computation of the gain in a metric. The proposed framework introduces a variant of online metrics called pMetrics (short for personalized metrics) that are based on the average search habits of users for the relevance signal of interest. Through extensive online experiments on a large population of GMail search users, we show that pMetrics are effective in terms of their sensitivity, robustness, and stability compared to their standard variants as well as baselines with different normalization factors.

Weather affects our mood and behavior, and through them, many aspects of our life. When it is sunny, people become happier and smile, but when it rains, some get depressed. Despite this evidence and the abundance of weather data, weather has mostly been overlooked in the machine learning and data science research. This work shows how causal analysis can be applied to discover the effects of weather on TV watching patterns and how it can be applied for user modeling. We make several contributions. First, we show that some weather attributes, e.g., pressure and precipitation, cause significant changes in TV watching patterns. Second, we compare the results obtained for different levels of user granularity and different types of users. This showcases that causal analysis can be a valuable tool in user modeling. To the best of our knowledge, this is the first large-scale causal study of the impact of weather on TV watching patterns.

The majority of Internet traffic is now dominated by streamed video content. As video quality continues to increase, the strain that streaming traffic places on the network infrastructure also increases. Caching content closer to users, e.g., using Content Distribution Networks, is a common solution to reduce the load on the network. A simple approach to selecting what to put in regional caches is to put the videos that are most popular globally across the entire customer base. However, this approach ignores distinct regional taste. In this paper we explore the question of how a video content provider could go about determining whether or not they should use a cache filling policy based solely upon global popularity or take into account regional tastes as well. We propose a model that captures the overlap between inter-regional and intra-regional preferences. We focus on movie content and derive a synthetic model that captures “taste” using matrix factorization, similarly to the method used in recommender systems. Our model enables us to widely explore the parameter space, and derive a set of metrics providers can use to determine whether populating caches according to regional of global tastes provides better cache performance.

Easy accessibility can often lead to over-consumption, as seen in food and alcohol habits. On video on-demand (VOD) services, this has recently been referred to as binge watching, where potentially entire seasons of TV shows are consumed in a single viewing session. While a user viewership model may reveal this binging behavior, creating an accurate model has several challenges, including censored data, deviations in the population, and the need to consider external influences on consumption habits. In this paper, we introduce a novel statistical mixture model that incorporates these factors and presents a first of its kind characterization of viewer consumption behavior using a real-world dataset that includes playback data from a VOD service. From our modeling, we tackle various predictive tasks to infer the consumption decisions of a user in a viewing session, including estimating the number of episodes they watch and classifying if they continue watching another episode. Using these insights, we then identify binge watching sessions based on deviation from normal viewing behavior. We observe different types of binging behavior, that binge watchers often view certain content out-of-order, and that binge watching is not a consistent behavior among our users. These insights and our findings have application in VOD revenue generation, consumer health applications, and customer retention analysis.

A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to constraints, and then observes stochastic weights of these items and receives their sum as a payoff. In this paper, we close the problem of computationally and sample efficient learning in stochastic combinatorial semi-bandits. In particular, we analyze a UCB-like algorithm for solving the problem, which is known to be computationally efficient; and prove O(K L (1 / ∆) \log n) and O(\sqrtK L n \log n) upper bounds on its n-step regret, where L is the number of ground items, K is the maximum number of chosen items, and ∆is the gap between the expected returns of the optimal and best suboptimal solutions. The gap-dependent bound is tight up to a constant factor and the gap-free bound is tight up to a polylogarithmic factor.

A search engine usually outputs a list of K web pages. The user examines this list, from the first web page to the last, and chooses the first attractive page. This model of user behavior is known as the cascade model. In this paper, we propose cascading bandits, a learning variant of the cascade model where the objective is to identify K most attractive items. We formulate our problem as a stochastic combinatorial partial monitoring problem. We propose two algorithms for solving it, CascadeUCB1 and CascadeKL-UCB. We also prove gap-dependent upper bounds on the regret of these algorithms and derive a lower bound on the regret in cascading bandits. The lower bound matches the upper bound of CascadeKL-UCB up to a logarithmic factor. We experiment with our algorithms on several problems. The algorithms perform surprisingly well even when our modeling assumptions are violated.

A stochastic combinatorial semi-bandit is an online learning problem where at each step a learning agent chooses a subset of ground items subject to combinatorial constraints, and then observes stochastic weights of these items and receives their sum as a payoff. In this paper, we consider efficient learning in large-scale combinatorial semi-bandits with linear generalization, and as a solution, propose two learning algorithms called Combinatorial Linear Thompson Sampling (CombLinTS) and Combinatorial Linear UCB (CombLinUCB). Both algorithms are computationally efficient as long as the offline version of the combinatorial problem can be solved efficiently. We establish that CombLinTS and CombLinUCB are also provably statistically efficient under reasonable assumptions, by developing regret bounds that are independent of the problem scale (number of items) and sublinear in time. We also evaluate CombLinTS on a variety of problems with thousands of items. Our experiment results demonstrate that CombLinTS is scalable, robust to the choice of algorithm parameters, and significantly outperforms the best of our baselines.

We propose combinatorial cascading bandits, a class of partial monitoring problems where at each step a learning agent chooses a tuple of ground items subject to constraints and receives a reward if and only if the weights of all chosen items are one. The weights of the items are binary, stochastic, and drawn independently of each other. The agent observes the index of the first chosen item whose weight is zero. This observation model arises in network routing, for instance, where the learning agent may only observe the first link in the routing path which is down, and blocks the path. We propose a UCB-like algorithm for solving our problems, CombCascade; and prove gap-dependent and gap-free upper bounds on its n-step regret. Our proofs build on recent work in stochastic combinatorial semi-bandits but also address two novel challenges of our setting, a non-linear reward function and partial observability. We evaluate CombCascade on two real-world problems and show that it performs well even when our modeling assumptions are violated. We also demonstrate that our setting requires a new learning algorithm.

An online advertisement’s clickthrough rate provides a fundamental measure of its quality, which is widely used in ad selection strategies. Unfortunately, ads placed in contexts where they are rarely viewed—or where users are unlikely to be interested in commercial results—may receive few clicks regardless of their quality. In this article, we model the variability of a user’s browsing behavior for the purpose of click analysis and prediction in sponsored search. Our model incorporates several important contextual factors that influence ad clickthrough rates, including the user’s query and ad placement on search engine result pages. We formally model these factors with respect to the list of ads displayed on a result page, the probability that the user will initiate browsing of this list, and the persistence of the user in browsing the list. We incorporate these factors into existing click models by augmenting them with appropriate query and location biases. Using expectation maximization, we learn the parameters of these augmented models from click signals recorded in the logs of a commercial search engine. To evaluate the performance of the models and to compare them with state-of-the-art performance, we apply standard evaluation metrics, including log-likelihood and perplexity. Our evaluation results indicate that, through the incorporation of query and location biases, significant improvements can be achieved in predicting browsing and click behavior in sponsored search. In addition, we explore the extent to which these biases actually reflect varying behavioral patterns. Our observations confirm that correlations exist between the biases and user search behavior.

Clickthrough rate provides a fundamental measure of advertising quality, which is widely used in ad selection strategies. However, ads placed in contexts where they are rarely viewed, or where users are unlikely to be interested in commercial results, may receive few clicks regardless of their quality. In this paper, we gain insight into user browsing and click behavior for the purpose of click analysis in sponsored search domain. The list of ads displayed on a page, the user's initial motivation to browse this list, and the persistence of the user are among the contextual factors considered in this paper. We propose a probabilistic model for user's browsing and click behavior using these contextual factors. To evaluate the performance of the model, we compare it with state-of-the-art methods. The experimental results confirm that these contextual factors can better reflect user browsing and click behavior in sponsored search.

Traditional editorial effectiveness measures, such as nDCG, remain standard for Web search evaluation. Unfortunately, these traditional measures can inappropriately reward redundant information and can fail to reflect the broad range of user needs that can underlie a Web query. To address these deficiencies, several researchers have recently proposed effectiveness measures for novelty and diversity. Many of these measures are based on simple cascade models of user behavior, which operate by considering the relationship between successive elements of a result list. The properties of these measures are still poorly understood, and it is not clear from prior research that they work as intended. In this paper we examine the properties and performance of cascade measures with the goal of validating them as tools for measuring effectiveness. We explore their commonalities and differences, placing them in a unified framework; we discuss their theoretical difficulties and limitations, and compare the measures experimentally, contrasting them against traditional measures and against other approaches to measuring novelty. Data collected by the TREC 2009 Web Track is used as the basis for our experimental comparison. Our results indicate that these measures reward systems that achieve an balance between novelty and overall precision in their result lists, as intended. Nonetheless, other measures provide insights not captured by the cascade measures, and we suggest that future evaluation efforts continue to report a variety of measures.

The Maximum Entropy Method provides one technique for validating search engine effectiveness measures. Under this method, the value of an effectiveness measure is used as a constraint to estimate the most likely distribution of relevant documents under a maximum entropy assumption. This inferred distribution may then be compared to the actual distribution to quantify the "informativeness" of the measure. The inferred distribution may also be used to estimate values for other effectiveness measures. Previous work focused on traditional effectiveness measures, such as average precision. In this paper, we extend the Maximum Entropy Method to the newer cascade and intent-aware effectiveness measures by considering the dependency of the documents ranked in a results list. These measures are intended to reflect the novelty and diversity of search results in addition to the traditional relevance. Our results indicate that intent-aware measures based on the cascade model are informative in terms of both inferring actual distribution and predicting the values of other retrieval measures.

Understanding the intent underlying user's queries may help personalize search results and therefore improve user satisfaction. We develop a methodology for using the content of search engine result pages (SERPs) along with the information obtained from query strings to study characteristics of query intent, with a particular focus on sponsored search. This work represents an initial step towards the development and evaluation of an ontology for commercial search, considering queries that reference specific products, brands and retailers. The characteristics of query categories are studied with respect to aggregated user's clickthrough behavior on advertising links. We present a model for clickthrough behavior that considers the influence of such factors as the location of ads and the rank of ads, along with query category. We evaluate our work using a large corpus of clickthrough data obtained from a major commercial search engine. Our findings suggest that query based features, along with the content of SERPs, are effective in detecting query intent. The clickthrough behavior is found to be consistent with the classification for the general categories of query intent, while for product, brand and retailer categories, all is true to a lesser extent.

Despite the wide applicability of clustering methods, their evaluation remains a problem. In this paper, we present a metric for the evaluation of clustering methods. The data set to be clustered is viewed as a sample from a larger population, with clustering quality measured in terms of our predicted ability to discriminate between members of this population. We measure this property by training a classifier to recognize each cluster and measuring the accuracy of this classifier, normalized by a notion of expected accuracy. To demonstrate the applicability of this metric we apply it to Web queries. We investigated a commercially oriented data set of 1700 queries and a general data set of 4000 queries. Both sets are taken from the logs of a commercial Web search engine. Clustering is based on the contents of search engine result pages generated by executing the queries on the search engine from which they were taken. Multiple clustering algorithms are crossed with various weighting schemes to produce multiple clusterings of each query set. Our metric is used evaluate these clusterings. The results on the commercially oriented data set are compared to two pre-existing manual labelings, and are also used in an ad clickthrough experiment.

In this work, we investigate the contribution of query terms and their corresponding ad click rates on commercial intent of queries. A probabilistic model is proposed following the hypothesis that a query is likely to receive ad clicks based on contributions from its individual terms.

Clickthrough rate, bid, and cost-per-click are known to be among the factors that impact the rank of an ad shown on a search result page. Search engines can benefit from estimating ad clickthrough in order to determine the quality of ads and maximize their revenue. In this paper, a methodology is developed to estimate ad clickthrough rate by exploring user queries and clickthrough logs. As we demonstrate, the average ad clickthrough rate depends to a substantial extent on the rank position of ads and on the total number of ads displayed on the page. This observation is utilized by a baseline model to calculate the expected clickthrough rate for various ads. We further study the impact of query intent on the clickthrough rate, where query intent is predicted using a combination of query features and the content of search engine result pages. The baseline model and the query intent model are compared for the purpose of calculating the expected ad clickthrough rate. Our findings suggest that such factors as the rank of an ad, the number of ads displayed on the result page, and query intent are effective in estimating ad clickthrough rate.

We present a method for modeling, and automaticallyinferring, the current interest of a user in searchadvertising. Our task is complementary to that of predictingad relevance or commercial intent of a query in the aggregate, since the user intent may vary significantly for the same query. To achieve this goal, we develop a fine-grained user interaction model for inferring searcher receptiveness to advertising. We show that modeling the search context and behavior can significantly improve the accuracy of ad clickthrough prediction for the current user, compared to the existing state-of-the-artclassification methods that do not model this additional session level contextual and interaction information. In particular, our experiments over thousands of search sessions from hundreds of real users demonstrate that our model is more effective at predicting ad clickthrough within the same search session. Our work has other potential applications, such as improving searchinterface design (e.g., varying the number or type of ads) based on user interest, and behavioral targeting (e.g., identifying users interested in immediate purchase).

Evaluation measures act as objective functions to be optimized by information retrieval systems. Such objective functions must accurately reflect user requirements, particularly when tuning IR systems and learning ranking functions. Ambiguity in queries and redundancy in retrieved documents are poorly reflected by current evaluation measures. In this paper, we present a framework for evaluation that systematically rewards novelty and diversity. We develop this framework into a specific evaluation measure, based on cumulative gain. We demonstrate the feasibility of our approach using a test collection based on the TREC question answering track.