With the highly demanded requirements for manipulating large scientific datasets, scientists are in need of flexible cluster-level software to execute fast scientific data analysis. In this paper, we discuss whether the Apache Spark framework is suitable for scientific data management. We present our system SparkArray, which extends Spark with a multidimensional array data model and a set of common used array operations (e.g., filter, subarray, smooth and join). We present analysis and performance evaluation results on different implementation methods for executing array operations on clusters. We also compared the performance of SparkArray to SciDB, a recent scientific database management system, using the workloads of the Standard Science DBMS Benchmark (SS-DB). The results show that SparkArray is feasible alternative solution for large-scale scientific data management, especially when scientists require fast data loading or one-time data analysis on large scientific datasets.

Current major big data analytical stacks often consist of a general purpose, multi-staged computation framework (e.g. Hadoop) and an SQL query system (e.g. Hive) on its top. A key factor of query performance is the efficiency of data shuffling between two execution stages (e.g. Map/Reduce). In current data shuffling, various useful information about the shuffled data and the query on the data is simply wasted. In this paper, we make a strong case of cross-layer optimizations for Hive/Hadoop stack: we have designed and implemented a novel data shuffling mechanism in Hadoop, called Structured Data Shuffling (S-Shuffle), which carefully leverages the rich information in data and queries to optimize the overall query processing. Our experimental results with industry-standard TPC-H benchmark show that, by using S-Shuffle, the performance of SQL query processing on Hadoop can be improved by up to 2.4x.

Game tree search is a classical problem in the field of game theory and artificial intelligence. Fast game tree search algorithm is critical for computer games asking for real-time responses. In this paper, we focus on how to leverage massive parallelism capabilities of GPU to accelerate the speed of game tree search algorithms and propose a concise and general parallel game tree search algorithm on GPU. The performance model of our algorithm is presented and analyzed theoretically. We implement the algorithm for two real computer games called Connect6 and Chess. We also use these two games to verify the effectiveness and efficiency of our algorithm. Experiments support our theoretical results and show good performance of our approach. Compared to classical CPU-based game tree search algorithms, our algorithm can achieve speedups of 89.95x for Connect6 and 11.43x for Chess, in case of no pruning. When pruning is considered, which means the practical performance of our algorithm, the speedup can reach about 10.58x for Connect6 and 7.26x for Chess. The insight of our work is that using GPU is a feasible way to improve the performance of game tree search algorithms.

Equi-join is heavily used in MapReduce-based log processing. With the rapid growth of dataset sizes, join methods on MapReduce are extensively studied recently. We find that existing join methods usually cannot get high query performance and affordable storage consumption at the same time when faced with a huge amount of log data. They either only optimize one aspect but significantly sacrifice the other or have limited applications. In this paper, after analyzing characteristics of the workloads and underlying MapReduce, we present a join method with specific optimizations for log processing called RHJoin (Repartition Hash Join) and its implementation on Hadoop. In RHJoin, reference tables are partitioned in the pre-processing step, the log table is partitioned on the map side and hash join is executed on the reduce side. The shuffle procedure of MapReduce is also optimized by removing the sort step and overlapping the execution of mappers and reducers. Comprehensive experiments show that RHJoin achieves high query performance with only a small extra storage cost, and has wide application circumstances for log processing.

It is very important to evaluate the MapReduce-based frameworks for scientific data processing applications. Scientists need a low-cost, scalable, easy-to-use and fault-tolerance platform for large volume data processing eagerly. This paper presents an implementation of a scientific data management benchmark, SSDB, on Hive, a MapReduce-based data warehouse. A complete strategy of migrating SSDB to Hive is described in detail including query HQL implementation, data partition schema and adjustments of underlying storage facilities. We have tuned the performance using several system parameters provided by Hive, Hadoop and HDFS. This paper provides preliminary results and analysis. Evaluation results indicate that Hive achieves acceptable performance for some data analysis tasks even compared with some high efficient distributed parallel databases, but it needs subtle adjustments of underlying storage facilities and indexing mechanism.

Game tree search is a classical problem in the field of game theory and artificial intelligence. Fast game tree algorithm is critical for computer games asking for real-time responses. In this paper, we focus on how to leverage massive parallelism capabilities of GPUs to accelerate the speed of game tree algorithms and propose a concise and general parallel game tree algorithm on GPUs. The performance model of the algorithm is presented and analyzed theoretically. We also implement the algorithm for a real computer game called Connect6 and use it to verify the effectiveness and efficiency of our algorithm. Experiments support our theoretical results and show good performance of our approach. Compared to classical CPU-based game tree algorithms, our algorithm can achieve speedup of 70.8 in case of no pruning. When pruning is considered (which means the practical performance of our algorithm), the speedup can reach about 7.0. The insight of our work is that using GPUs is a feasible way to improve the performance of game tree algorithms.

In this paper, we design and implement IRain, a prototype of storage cloud for computer scientists and graduate students to manage personal data that spreads over the web. IRain (1) integrates personal data with various metainfo structures that comes from different web sites and personal computers; (2) provides a global, unified environment to users and supports user-defined file views via flexible combination of tags; (3) offers an easy way to integrate new web services via a VFS-liked interface.

Distributed applications are increasingly being built by Web services. This paper explores a dynamic service execution technique on virtual machines to exploit parallelism of service-based applications. The Dynamic service execution technique is able to determine data dependence of service invocations at run-time and executes independent service invocations concurrently with the help of stateless property of Web services. Based on this technique, we design and implement a virtual machine-based runtime environment called Abacus Virtual Machine (AVM). On AVM, service-based applications are programmed in a conventional, sequential programming model, which involves no extra parallel programming complexity. At the same time AVM also need to make balancing between the goal of exploiting parallelism and the cost brought by the dynamic service execution. Our experiments show that the cost is no more than 9% of CPU time and 10% memory cost. We believe that the penalty is acceptable for the benefits that AVM provides.

SOAP protocol has emerged as the Web service communication standard. Because of the relatively poor performance, many researchers focus on improving the speed of processing SOAP message. In this paper, we propose SPI, which introduces client usage pattern to the low level SOAP process infrastructure, in order to improve the performance of some kind of Web services applications with specific communication patterns. The pack interface of SPI is an approach to reduce the number of the SOAP messages to improve the latency on client side. This SOAP optimization technique packs concurrent service requests into one SOAP message on client side, and unpacks and executes them on server side. Our implementation and performance study demonstrates the usefulness and effectiveness of the technique

Grid computing is a new paradigm for distributed computing, and service has become building block of grid applications. However, current approaches can not free developers from low-level laborious work when building grid applications. We propose a service-oriented virtual machine called Abacus Virtual Machine to simplify the task of grid application development. As a language level virtual machine, it provides a service-oriented instruction set to abstract the operations on the services of a grid application. It also virtualizes services and creates a virtual global system image for grid applications, thus services can be transparently distributed and shared. In this way, Abacus Virtual Machine hides the cumbersome underlying details from programmers and reduces the complexity greatly in grid application development