There has been considerable work on user browsing models for search engine results, both organic and sponsored. The click-through rate (CTR) of a result is the product of the probability of examination (will the user look at the result) times the perceived relevance of the result (probability of a click given examination). Past papers have assumed that when the CTR of a result varies based on the pattern of clicks in prior positions, this variation is solely due to changes in the probability of examination.

Statistical inference for massive graphs is a challenging problem since it is hard to scale the existing statistical models to such settings. However, in many applications one is often interested in understanding the interaction of a single node with the rest of the graph. Recently, a data structure called Communities of Interest was introduced in the literature that represent a graph as a union of small subgraphs, each centered on a fixed node. In this paper, we describe statistical models that could be fitted to these subgraphs and are extensions of Stochastic Blockmodels that exist in the social networks literature. Our method can handle sparseness using shrinkage estimation, weights on edges and could be fitted to a graph consisting of around 100–200 nodes in less than 2 minutes on a 1 GHz PC. The methods are exemplified through a Community of Interest of a customer who has signed up for some service with AT&T.

Massive transaction streams present a number of opportunities for data mining techniques. The transactions in such streams might represent calls on a telephone network, commercial credit card purchases, stock market trades, or HTTP requests to a web server. While historically such data have been collected for billing or security purposes, they are now being used to discover how the transactors, for example, credit-card numbers or IP addresses, use the associated services.Over the past 5 years, we have computed evolving profiles (called signatures ) of transactors in several very large data streams. The signature for each transactor captures the salient features of his or her behavior through time. Programs for processing signatures must be highly optimized because of the size of the data stream (several gigabytes per day) and the number of signatures to maintain (hundreds of millions). Originally, we wrote such programs directly in C, but because these programs often sacrificed readability for performance, they were difficult to verify and maintain.Hancock is a domain-specific language we created to express computationally efficient signature programs cleanly. In this paper, we describe the obstacles to computing signatures from massive streams and explain how Hancock addresses these problems. For expository purposes, we present Hancock using a running example from the telecommunications industry; however, the language itself is general and applies equally well to other data sources.

This paper considers the framework of the so-called "market basket problem", in which a database of transactions is mined for the occurrence of unusually frequent item sets. In our case, "unusually frequent" involves estimates of the frequency of each item set divided by a baseline frequency computed as if items occurred independently. The focus is on obtaining reliable estimates of this measure of interestingness for all item sets, even item sets with relatively low frequencies. For example, in a medical database of patient histories, unusual item sets including the item "patient death" (or other serious adverse event) might hopefully be flagged with as few as 5 or 10 occurrences of the item set, it being unacceptable to require that item sets occur in as many as 0.1% of millions of patient reports before the data mining algorithm detects a signal. Similar considerations apply in fraud detection applications. Thus we abandon the requirement that interesting item sets must contain a re...

Article Hancock: a language for extracting signatures from data streams Share on Authors: Corinna Cortes AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJ AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJView Profile , Kathleen Fisher AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJ AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJView Profile , Daryl Pregibon AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJ AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJView Profile , Anne Rogers AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJ AT&T Labs-Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJView Profile Authors Info & Claims KDD '00: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data miningAugust 2000 Pages 9–17https://doi.org/10.1145/347090.347094Online:01 August 2000Publication History 102citation888DownloadsMetricsTotal Citations102Total Downloads888Last 12 Months4Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Experiencehas shown that the data extraction, parsing, cleaning and analysis steps of a KDD problem account for a much larger expenditure of resources (time and money) than the statistical modeling or machine learning part.Couple this statement with the need for fast turn-around time in commercial applications, and the obvious conclusion is that it is impractical to start from scratch for each new KDD application.To deal with this situation, we propose the use of an information mining platform that amortizes several of the critical pre-and post-processing steps needed to apply KDD.Thus, new KDD applications can leverage the platform for efficiency and robustness.

A feature of data mining that distinguishes it from "classical" machine learning (ML) and statistical modeling (SM) is scale. The community seems to agree on this yet progress to this point has been limited. We present a methodology that addresses scale in a novel fashion that has the potential for revolutionizing the field. While the methodology applies most directly to flat (row by column) data sets we believe that it can be adapted to other representations. Our approach to the problem is not to scale up individual ML and SM methods. Rather we prefer to leverage the entire collection of existing methods by scaling down the data set. We call the method squashing. Our method demonstrably outperforms random sampling and a theoretical argument suggests how and why it works well. Squashing consists of three modular steps: grouping, momentizing, and generating (GMG). These three steps describe the squashing pipeline whereby the original (very large data set) is sectioned off into mutual...

This talk is an interim report on the 5 year plan launched in 1996 to provide a theoretical and computational foundation of Statistics for massive data sets. The plan coincided with the formation of AT&T Labs and the proposed research agenda of the InfoLab, which is both a physical laboratory and an interdisciplinary collection of information researchers in CS, mathematics, and Statistics. At the halfway point of this odyssey we can identify some success stories but more importantly it is an opportune time to re-calibrate the challenges and the milestones.