We present GALGO, a system for large scale graph analytics. GALGO provides complex graph analytics in a parallel cluster, exploiting a parallel database system as a computation engine. In this demonstration we show that fundamental graph algorithms including all pairs shortest path, single source shortest path, PageRank, triangle counting, connected components and reachability can be solved completely with queries, dynamically generated by our system. Our system presents performance that is very competitive to state-of-the-art graph systems. Furthermore, our out-of-core graph computation can process graphs larger than available main memory, without compromising performance.

In a typical Data Science project, the analyst uses many programming languages to explore and analyze big data coming from diverse data sources. A major challenge is managing and pre-processing so much data, with potentially inconsistent content, significant redundancy, in diverse formats, with varying data quality. Database systems research has tackled such problems for a long time, but mostly on relational databases. With such motivation in mind, this paper compares strengths and weaknesses of popular languages used nowadays from a database pespective: Python, R and SQL. We discuss the entire analytic pipeline, going from data integration, cleaning and pre-processing to model application and tuning. From a database systems perspective, we present a comprehensive survey of storage mechanisms, data processing algorithms, external algorithms, run-time memory management, consistency, optimizations and parallel processing. From a programming languages angle, we consider elegance, expressiveness, abstraction, composability, interactive behavior and automatic code optimization. We present a short experimental evaluation comparing the performance of the three languages on typical data exploration and pre-processing tasks. Our conclusion: there is no winner.

Big Data Systems (Hadoop, DBMSs) require a complicated setup and tuning to store and process big data on a parallel cluster. This is mainly due to static partitioning when data sets are loaded or copied into the file system. Parallel processing thereafter works in a distributed manner, aiming for balanced parallel execution across nodes. Node synchronization, data redistribution and distributed caching in main memory are difficult to tune in the system. On the other hand, there exist analytical problems and algorithms, which can be computed in parallel, with minimal synchronization and fully independent computation. Moreover, some problems can be solved in one pass or few passes. In this paper, we introduce a low cost, yet useful, processing architecture in which data sets are dynamically partitioned at run-time and storage is transient. Each node processes one partition independently and partial results are gathered at the master processing node. Surprisingly, we show this architecture works well for some popular machine learning models as well as some graph algorithms. We attempt to identify which problem characteristics enable such efficient processing, and we also show the main bottleneck is the initial data set partitioning and distribution across nodes. We anticipate our architecture can benefit parallel processing in the cloud, where a dynamic number of virtual processors is decided at runtime or when the data set is analyzed for a short time.

Analyzing large data sets are challenging. Most data analytics research has proposed parallel algorithms that outside a DBMS because SQL is considered inadequate for complexity computations. R and Python are popular analysis systems that provide a vast collection of mathematical models and functions. However, they are limited by main memory and single computer. Recently, parallel DBMSs have significantly improved query processing performance. Moreover, SQL queries are elegant and efficient. This paper introduces a novel system architecture integrating a popular analysis system and parallel DBMSs, which has the matrix multiplication involving a large matrix evaluated inside a parallel DBMS and complex mathematical computations are done in R or Python. Many graph problems can be solved by matrix multiplication. In this paper, we show optimized queries which perform matrix multiplication in DBMSs to solve two fundamental graph problems, single-source reachability and transitive closure.

In big data analytics, graph problems are as important as machine learning. There exist many algorithms to analyze large graphs, but most of them are limited by the main memory. On the other hand, a lot of data stored on DBMSs needs to be analyzed as graphs. Even data is not in DBMSs, it is fast to load large data sets into DBMSs. More importantly, DBMSs can work in parallel and they do not have RAM limitations. Based on these reasons, we propose several algorithms that compute metrics and properties of the graph as well as help us to understand the graph structure specifically diameter, betweenness centrality. This work is a big step beyond transitive closure and recursive queries. We propose SQL queries that can work on a large graph stored in relational form as triples. And we prove these queries can compute complex graph metrics in a more flexible and efficient manner. Moreover, we study how to optimize the SQL queries combining demanding joins and aggregations that remove the main memory limitation and also can work in parallel. We provide an experimental evaluation to evaluate the accuracy and performance of our SQL algorithms by comparing our algorithms with popular platforms including Python and Spark. In general, the experiments show our SQL algorithms are accurate and efficient, without memory limitation.