We focus on the problem of detecting anomalous run-time behavior of distributed applications from their execution logs. Specifically we mine templates and template sequences from logs to form a control flow graph (cfg) spanning distributed components. This cfg represents the baseline healthy system state and is used to flag deviations from the expected behavior of runtime logs. The novelty in our work stems from the new techniques employed to: (1) overcome the instrumentation requirements or application specific assumptions made in prior log mining approaches, (2) improve the accuracy of mined templates and the cfg in the presence of long parameters and high amount of interleaving respectively, and (3) improve by orders of magnitude the scalability of the cfg mining process in terms of volume of log data that can be processed per day. We evaluate our approach using (a) synthetic log traces and (b) multiple real-world log datasets collected at different layers of application stack. Results demonstrate that our template mining, cfg mining, and anomaly detection algorithms have high accuracy. The distributed implementation of our pipeline is highly scalable and has more than 500 GB/day of log data processing capability even on a 10 low-end VM based (Spark + Hadoop) cluster. We also demonstrate the efficacy of our end-to-end system using a case study with the Openstack VM provisioning system.

Time-travel text search over a temporally evolving document collection is useful in various applications. Supporting a wide range of query classes demanded by these applications require different index layouts optimized for their respective query access patterns. The problem we tackle is how to efficiently handle different query classes using the same index layout.

High-resolution MALDI-TOF (matrix-assisted laser desorption/ionization time-of-flight) mass spectrometry has shown promise as a screening tool for detecting discriminatory protein patterns. The major computational obstacle in analyzing MALDI-TOF data is a large number of mass/charge peaks (a.k.a. features, data points). With the number of data points easily going beyond one million for a single sample, efficient feature selection is critical for unequivocal protein pattern discovery. To tackle this problem, we have developed a multi-step strategy for data preprocessing and afterwards feature selection. The preprocessing is composed of binning, baseline correction, and normalization. For the preprocessed data, we propose a new feature subset selection method that is a hybrid filter/wrapper approach. Based on the two feature subsets for each feature, high and low correlated subsets, a feature is assigned a weight which indicates the extent of feature importance. Our scheme is applied to the analysis of labor dataset to predict delivery time of pregnant women. To validate the performance of the proposed algorithm, experiments are performed in comparison with other feature selection and classification methods. We show that our proposed approach outperforms other algorithms

Ovarian cancer recurs at the rate of 75% within a few months or several years later after therapy. Early recurrence, though responding better to treatment, is difficult to detect. Recently, high-resolution MALDI-TOF (matrix-assisted laser desorption/ionization time-of-flight) mass spectrometry has shown promise as a screening tool for detecting discriminatory protein patterns. The major computational obstacle in analyzing MALDI-TOF data is a large number of mass/charge peaks (a.k.a. features, data points). To tackle this problem, we have developed a multi-step strategy for data preprocessing and afterwards feature selection. The preprocessing is composed of binning, baseline correction, and normalization. For the preprocessed data, we propose a new feature subset selection method. Our scheme is applied to the analysis of ovarian cancer dataset to predict early relapse in ovarian cancer. To validate the performance of the proposed algorithm, experiments are performed in comparison with other feature selection and classification methods. We show that our proposed approach outperforms other algorithms

Understanding the characteristics of the Internet delay space (i.e., the all-pairs set of static round-trip propagation delays among edge networks in the Internet) is important for the design of global-scale distributed systems. For instance, algorithms used in overlay networks are often sensitive to violations of the triangle inequality and to the growth properties within the Internet delay space. Since designers of distributed systems often rely on simulation and emulation to study design alternatives, they need a realistic model of the Internet delay space.Our analysis shows that existing models do not adequately capture important properties of the Internet delay space. In this paper, we analyze measured delays among thousands of Internet edge networks and identify key properties that are important for distributed system design. Furthermore, we derive a simple model of the Internet delay space based on our analytical findings. This model preserves the relevant metrics far better than existing models, allows for a compact representation, and can be used to synthesize delay data for simulations and emulations at a scale where direct measurement and storage are impractical.

In tree-based multicast systems, a relatively small number of interior nodes carry the load of forwarding multicast messages. This works well when the interior nodes are highly-available, dedicated infrastructure routers but it poses a problem for application-level multicast in peer-to-peer systems. SplitStream addresses this problem by striping the content across a forest of interior-node-disjoint multicast trees that distributes the forwarding load among all participating peers. For example, it is possible to construct efficient SplitStream forests in which each peer contributes only as much forwarding bandwidth as it receives. Furthermore, with appropriate content encodings, SplitStream is highly robust to failures because a node failure causes the loss of a single stripe on average. We present the design and implementation of SplitStream and show experimental results obtained on an Internet testbed and via large-scale network simulation. The results show that SplitStream distributes the forwarding load among all peers and can accommodate peers with different bandwidth capacities while imposing low overhead for forest construction and maintenance.