The social sciences strive to understand the political, social, and cultural world around us, but have been impaired by limited access to the quantitative data sources enjoyed by the hard sciences. Careful analysis of Web document streams holds enormous potential to solve longstanding problems in a variety of social science disciplines through massive data analysis. This paper introduces the TextMap Access system, which provides ready access to a wealth of interesting statistics on millions of people, places, and things across a number of interesting web corpora. Powered by a flexible and scalable distributed statistics computation framework using Hadoop, continually updated corpora include newspapers, blogs, patent records, legal documents, and scientific abstracts; well over a terabyte of raw text and growing daily. The Lydia Textmap Access system, available through http://www.textmap.com/access, provides instant access for students and scholars through a convenient web user-interface. We describe the architecture of the TextMap Access system, and its impact on current research in political science, sociology, and business/marketing.

The problem of ethnicity identification from names has a variety of important applications, including biomedical research, demographic studies, and marketing. Here we report on the development of an ethnicity classifier where all training data is extracted from public, non-confidential (and hence somewhat unreliable) sources. Our classifier uses hidden Markov models (HMMs) and decision trees to classify names into 13 cultural/ethnic groups with individual group accuracy comparable accuracy to earlier binary (e.g., Spanish/non-Spanish) classifiers. We have applied this classifier to over 20 million names from a large-scale news corpus, identifying interesting temporal and spatial trends on the representation of particular cultural/ethnic groups.

Formal language constrained shortest path problems are concerned with finding shortest paths in labeled graphs. These labeled paths have the constraint that the concatenation of labels along a path constitute a valid string in some formal language \Lambda over alphabet \Sigma. These problems are well studied where the formal language is regular or context-free, and have been used in a variety of applications ranging from databases, to transportation planning, to programming languages. Barrett, Jacob, and Marathe's best algorithm for the context-free language constrained path problem runs in O(|V|^3|N| |P|) time, where N is the set of non-terminals for the input grammar and P is the set of productions (expressed in Chomsky Normal Form). We present a work and time efficient distributed version of this algorithm that may be distributed on up to O(|V| |N|) nodes.

This paper proposes using collisions of Pareto random variables in traffic analysis and in generating fictitious network traffic that follows various Pareto distributions. Pareto distributions are commonly found in network statistics, but the distributions may be truncated or overlapping, thus making it hard to estimate their sample parameters. Therefore, this paper investigates methods of computing parameters of binned collisions of Pareto random variables. This paper explores an indicator variable approach to analyzing collisions of Pareto random variables. These collisions are initially modeled by the birthday problem or paradox and then they are extended to understand independence of collisions. This paper's use of indicator variables simplifies the calculation of higher moments for binned collisions of Pareto random variables