Graph database (DB) systems are increasing their popularity in big-data analysis and machine learning particularly in the areas of e-commerce recommendation, fraud detection, and social-media analytics. Speed-up and spatial scalability of their DB transactions are pursued with various techniques such as index-free access to graph components in Neo4j, graph sharding over a cluster system in ArangoDB, and graph DB construction over distributed memory in AnzoGraph. However, these techniques have their respective challenges: difficulty in expanding an index-free graph over distributed memory, slow-down in accessing distributed disks, and a bottleneck incurred by repetitive master-to-worker distributions of query pipelines.As a solution to these problems, we are applying multi-agent technologies to distributed graph DB construction: multiple user processes over a cluster system maintain portion of distributed graph in their cache space; their cache contents are synchronized through a software-snooped write-back and write-update protocol; and a DB user from any cluster node dispatches an agent that handles an independent graph query through navigating over distributed graph. To follow current trends in graph DB standardization, we adopt the Cypher language whose queries are translated into agent code.This paper presents a new distributed hash-map implementation and its application to our graph DB system; differentiates it from Hazelcast from the viewpoints of its memory coherency and access speed; describes our translator that generates agent code from Cypher queries; and examines graph DB creation and traversal performance of agents in comparison with Neo4j and ArangoDB.

Graph streaming has received substantial attention for the past 10+ years to cope with large-scale graph computation. Two major approaches, one using conventional data-streaming tools and the other accessing graph databases, facilitate continuous analysis of endlessly flowing graphs and query-based incremental construction of huge graphs, respectively. However, some scientific graphs including biological networks need to stay in memory for repetitive but various analyses. Although a cluster system, thus distributed memory can entirely handle a big graph in memory, a challenge is substantial overhead incurred by loading graphs into memory. A solution is hiding such graph-loading and construction overheads with graph computation in a pipelined fashion. We adapted this pipelining approach for agent-based graph computing where thousands of agents traverse a graph for finding its attributes and shape. We used the multi-agent spatial simulation (MASS) library to implement the concept. A huge graph is incrementally constructed in batches, each spawning and walking agents over the corresponding subgraph, and thus all eventually completing a given computation. We coded and ran two MASS benchmark programs: triangle counting and connected components, with which we evaluated our pipelined graph processing. The best performance was obtained once the batch size shrunk enough to fit cache memory, regardless of the number of cluster nodes. For a single node execution of connected components over a 140MB graph, our graph-pipelining implementation performed 7.7 times faster than non-pipelining execution. Its parallel execution with 24 cluster nodes achieved 8.3 times speed-up as compared to the pipelined single-node execution.

Some graph analyses, such as social network and biological network, need large-scale graph construction and maintenance over distributed memory space. Distributed data-streaming tools, including MapReduce and Spark, restrict some computational freedom of incremental graph modification and run-time graph visualization. Instead, we take an agent-based approach. We construct a graph from a scientific dataset in CSV, tab, and XML formats; dispatch many reactive agents on it; and analyze the graph in the form of their collective group behavior: propagation, flocking, and collision. The key to success is how to automate the run-time construction and visualization of agent-navigable graphs mapped over distributed memory. We implemented this distributed graph-computing support in the multi-agent spatial simulation (MASS) library, coupled with the Cytoscape graph visualization software. This paper presents the MASS implementation techniques and demonstrates its execution performance in comparison to MapReduce and Spark, using two benchmark programs: (1) an incremental construction of a complete graph and (2) a KD tree construction.