We design and implement a scalable version of loopy belief propagation (BP), a widely used algorithm for performing inference on probabilistic graphical models. However, implementations of BP on generic data processing platforms such as Apache Spark do not scale well to very large graphical models containing billions of vertices. To handle such large-scale graphs, we leverage a number of strategies. Our implementation is based on Apache Spark GraphX. We propose a novel graph partitioning strategy to reduce both computation and communication overhead providing a 2x speed-up. We use efficient memory management for storing the graph and shared memory for highspeed communication. To evaluate performance and demonstrate scalability of the approach, we perform a range of experiments including using real-world graphs with billions of vertices, where we achieve an overall 10x speed-up over a vanilla Spark baseline. Further, we apply our BP implementation to infer the probability of a website being malicious by performing inference on a graphical model derived from real, large-scale hyperlinked webcrawl data. We have open sourced our implementation.

Given the growing availability of affordable scale-up servers, our goal is to bring the performance benefits of in-memory processing on scale-up servers to an increasingly common class of data analytics applications that process small to medium size datasets (up to a few 100GBs) that can easily fit in the memory of a typical scale-up server To achieve this goal, we leverage Spark, an existing memory-centric data analytics framework with wide-spread adoption among data scientists. Bringing Spark's data analytic capabilities to a scale-up system requires rethinking the original design assumptions, which, although effective for a scale-out system, are a poor match to a scale-up system resulting in unnecessary communication and memory inefficiencies.

Present day machine learning is computationally intensive and processes large amounts of data. It is implemented in a distributed fashion in order to address these scalability issues. The work is parallelized across a number of computing nodes. It is usually hard to estimate in advance how many nodes to use for a particular workload. We propose a simple framework for estimating the scalability of distributed machine learning algorithms. We measure the scalability by means of the speedup an algorithm achieves with more nodes. We propose time complexity models for gradient descent and graphical model inference. We validate the gradient descent model with experiments on deep learning training and graphical inferences with experiments on loopy belief propagation. The proposed framework was used to study the scalability of machine learning algorithms in Apache Spark.

We describe matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operations and shipping the matrix operations to be ran on the cluster, while keeping vector operations local to the driver. In the case of the Singular Value Decomposition, by taking this idea to an extreme, we are able to exploit the computational power of a cluster, while running code written decades ago for a single core. Another example is our Spark port of the popular TFOCS optimization package, originally built for MATLAB, which allows for solving Linear programs as well as a variety of other convex programs. We conclude with a comprehensive set of benchmarks for hardware accelerated matrix computations from the JVM, which is interesting in its own right, as many cluster programming frameworks use the JVM. The contributions described in this paper are already merged into Apache Spark and available on Spark installations by default, and commercially supported by a slew of companies which provide further services.

The need for fast on-line algorithms to analyze high data-rate measurements is a vital element in production settings. Given the ever-increasing number of data sources coupled with increasing complexity of applications, and workload patterns, anomaly detection methods should be light-weight and must operate in real-time. In many modern applications, data arrive in a streaming fashion. Therefore, the underlying assumption of classical methods that the data is a sample from a stable distribution is not valid, and Gaussian and non-parametric based methods such as the control chart and boxplot are inadequate. Streaming data is an ever-changing superposition of distributions. Detection of such changes in real-time is one of the fundamental challenges. We propose low-complexity robust modifications to the conventional Tukey boxplot based on fast highly efficient robust estimates of scale. Results using synthetic as well as real-world data show that our methods outperform the Tukey boxplot and methods based on Gaussian limits.

Nowadays we see an increasing number of global network attacks. These attacks are realized due to joint efforts of many distributed malicious software components (bots). It is very hard to investigate the effectiveness and efficiency of defense mechanisms against such attacks in practice. However these mechanisms might be simulated with the necessary fidelity. The paper outlines a framework and software tool intended for simulation of the Internet attacks and defense mechanisms against them. They are based on packet-level simulation and agent-oriented approach and intended to evaluate and compare different cooperative distributed defense mechanisms. The paper describes the simulation framework and software tool developed and their usage to analyze cooperative defense mechanisms against DDoS (Distributed Denial of Service) attacks. We investigate as mechanisms based on partial cooperation of distributed defense components, including DefCOM (Defensive Cooperative Overlay Mesh) and COSSACK (coordinated suppression of simultaneous attacks) as well as the approach based on full cooperation.

The paper presents an approach to design a cyberworld intended for the research of security systems functioning in the Internet. These systems are represented as the complex of various interacting teams of intelligent agents. The agents can be in the state of antagonistic counteraction or cooperation. The general conceptual model of agent teams' antagonistic counteraction and cooperation is suggested. The task of developing the agent cyberworld, where agents realize the distributed attacks and defense, is investigated on the example of "distributed denial of service" attack subject domain. The architecture and implementation of research environment for agent-oriented simulation is presented. The environment is implemented on the basis of discrete-event simulation system that allowed to combine the agent-oriented and network protocol based simulation

In the paper the approach for the simulation of counteraction between malefactors and defense systems in the Internet is considered. We try to model these antagonistic actors as software agents' teams. To simulate the teams' warfare it is proposed to use various computational models (from analytical and packet-based to virtual and testbeds). The main attention is drawn to the application of agent-oriented simulation based on the packet-based imitation of network security processes. Such approach provides acceptable fidelity and scalability in representing the attack and defense mechanisms. The approach is examined on an example of "distributed denial of service" attacks and defense simulation. We consider different phases of antagonistic teams' operations-learning, decision making and counteracting, including the adaptation of one team to the actions of opposite team