The Web abounds with dyadic data that keeps increasing by every single second. Previous work has repeatedly shown the usefulness of extracting the interaction structure inside dyadic data [21, 9, 8]. A commonly used tool in extracting the underlying structure is the matrix factorization, whose fame was further boosted in the Netflix challenge [26]. When we were trying to replicate the same success on real-world Web dyadic data, we were seriously challenged by the scalability of available tools. We therefore in this paper report our efforts on scaling up the nonnegative matrix factorization (NMF) technique. We show that by carefully partitioning the data and arranging the computations to maximize data locality and parallelism, factorizing a tens of millions by hundreds of millions matrix with billions of nonzero cells can be accomplished within tens of hours. This result effectively assures practitioners of the scalability of NMF on Web-scale dyadic data.

Map-Reduce is a programming model that enables easy development of scalable parallel applications to process a vast amount of data on large clusters of commodity machines. Through a simple interface with two functions, map and reduce, this model facilitates parallel implementation of many real-world tasks such as data processing jobs for search engines and machine learning.

In this paper1 we introduce the Holodex, a ‘holistic index’ for databases that includes a facility for statistics and aggregate-like computations. The Holodex is an integration of the conventional index and summarization over traversals of the index. It can store customized summaries in its data structure, and in this way it can maintain, and provide fast access to, summarized information. The Holodex rests on the Summary-Traversal Architecture a customizable summarization scheme for tree indexes. An important property of the summary-traversal architecture is that index structures defining an ordering on data can be augmented to provide extra summary information as well. For example, both tree indexes (such as the B+- Tree) and tree-hash hybrids (e.g., Multi-Level Trie Hashing and Interpolation Search Tree ) define an ordering, and they can be naturally extended to include summary information. This combination of indexing and summarization has a variety of uses, including computation of aggregate functions, rollups, bulk computation, and a variety of kinds of statistics, particularly those that are in some way related to order. More specifically, it is useful for computing nonparametric statistics - including rank statistics and order statistics - as well as direct implementation of queries like basic statistical tests on sample distributions.