Transaction data can arrive at a ferocious rate in the order that transactions are completed. The data contain an enormous amount of information about customers, not just transactions, but extracting up-to-date customer information from an ever changing stream of data and mining it in real-time is a challenge. This paper describes a statistically principled approach to designing short, accurate summaries or signatures of high dimensional customer behavior that can be kept current with a stream of transactions. A signature database can then be used for data mining and to provide approximate answers to many kinds of queries about current customers quickly and accurately, as an empirical study of the calling patterns of 96,000 wireless customers who made about 18 million wireless calls over a three month period shows.

Data-call records, internet packet headers, or other transaction records-are coming down a pipe at a ferocious rate, and we need to monitor statistics of the data.There is no reason to think that the data are normally distributed, so quantiles of the data are important to watch.The probe attached to the pipe has only limited memory, though, so it is impossible to compute the quantiles by sorting the data.The only possibility is to incrementally estimate the quantiles as the data fly by.This paper provides such an incremental quantile estimator.It resembles an exponentially weighted moving average in form, processing and memory requirements, but it is based on stochastic approximation so we call it an exponentially weighted stochastic approximation or EWSA.Simulations show that the EWSA outperforms other kinds of incremental estimates that also require minimal main memory, especially when extreme quantiles are tracked for patterns of behavior that change over time.Use of the EWSA is illustrated in an application to tracking call duration for a set of callers over a three month period.