Many vectorization languages and linear algebra libraries are often implemented as light-weight wrappers around high-performance array libraries, such as BLAS, and rely on the limited array storage structures and routines provided by these libraries, which are hard to extend and customize. In this paper, we describe a customizable framework for large-scale array programs in which arrays and array operations are abstract but their implementation is guided by user-defined storage mappings. We introduce a new storage structure for arrays, called a distributed tensor, which is a distributed collection of array blocks that may have any number of sparse and dense dimensions. In addition, we present rules for translating abstract array programs to high-performance distributed code that can run on Apache Spark. The performance of our system is on par with highly optimized linear algebra libraries, thus providing implementation independence and extensibility without sacrificing performance. Finally, we justify our claims by evaluating the performance of our system relative to Spark MLlib and TensorFlow.

Many graph algorithms can be expressed as repetitive computations that resemble matrix multiplication in which the addition and multiplication operations have been replaced with generalized operations that form an algebraic structure known as a semiring. Similar to matrix multiplication, these graph algorithms can be implemented in a distributed system using block arrays, which are distributed collections of non-overlapping dense arrays. In a distributed system, computations on block matrices are significantly faster than the same computations in the coordinate format in terms of computation and communication cost. Given the rise of Big Data and data-driven decision-making, our goal is to rewrite sequential programs on large graphs that are initially written to run on a single computer using loops and standard array operations to data-parallel programs on array blocks that can run on a distributed system. Our framework, called OSQLgen, automatically parallelizes graph programs with loops and arrays to distributed data-parallel programs. OSQLgen recognizes programs that are equivalent to a semiring structure and translates them to fast Spark SQL programs on block arrays that can run on a computer cluster in a distributed mode. We compare the performance of OSQLgen with GraphX, GraphFrames, and hand-written Spark SQL programs on coordinate and block arrays on various graph problems. On certain graph problems, OSQLgen is up to 36x faster than GraphX, 25x faster than GraphFrames, and up to 99x faster than hand-written Spark SQL programs on coordinate arrays, giving performance close to that of hand-written Spark SQL programs on block arrays.

Many programs written to analyze data are expressed in terms of array operations in an imperative programming language with loops. However, for data analysts who need to analyze vast volumes of data, large-scale data-intensive processing is becoming a necessity. Hence, they want to convert their programs, originally written to run on a single computer, to work on current Big Data systems, such as Map-Reduce and Spark, so that they can process larger amounts of data. We present a novel framework, called SQLgen, that automatically translates imperative programs with loops and array operations to distributed data-parallel programs. Unlike related work, SQL- gen translates these programs to SQL, which can be translated to more efficient code since it can be optimized using a relational database optimizer. SQLgen has been implemented on Spark SQL. We compare the performance of SQLgen with DIABLO, hand-written RDD-based, and Spark SQL programs on real- world problems. SQLgen is up to 78× faster than DIABLO and up to 25× faster than hand-written RDD-based programs, giving performance close to that of hand-written programs in Spark SQL.

Large volumes of data generated by scientific experiments and simulations come in the form of arrays, while programs that analyze these data are frequently expressed in terms of array operations in an imperative, loop-based language. But, as datasets grow larger, new frameworks in distributed Big Data analytics have become essential tools to large-scale scientific computing. Scientists, who are typically comfortable with numerical analysis tools but are not familiar with the intricacies of Big Data analytics, must now learn to convert their loop-based programs to distributed data-parallel programs. We present a novel framework for translating programs expressed as array-based loops to distributed data parallel programs that is more general and efficient than related work. We report on a prototype implementation on top of Spark and evaluate the performance of our system relative to hand-written programs.