To deliver business value, most data-driven enterprises and applications require data to be extracted and merged from otherwise siloed data storage platforms. FDC Cache has been designed and developed to enable the fusion and caching of data drawn from multiple small and/or Big Data stores. This capability executes a sequence of queries, wherein the results from one query may be used to constrain subsequent queries. The results of each query are linked with results from previous queries, incrementally building a cache of semantically linked data that can be used to support multiple independent data requests. FDC Cache uses Semantic Web technologies, and knowledge graphs in particular, to describe the relevant data and relationships in a computable model. This enables applications to reason over the graph, for example to dynamically retrieve targeted subsets of data comprised of previously disparate information. We have successfully applied FDC Cache to two distinct industrial use cases: (i) merging data across multiple sources to assemble information about current parts in a gas turbine, and (ii) dynamically aligning siloed data from electric grid transmission and distribution networks to an industry-standard common model, in which the cache creation time has been shown to scale sub-linearly with the number of data elements. FDC Cache has been open-sourced as part of the GE-developed open source Semantics Toolkit.

Additive technologies are expected to revolutionize manufacturing across almost every industry, but there is a sizable gap between the current state of the technology and the maturity required for it to achieve widespread adoption. Advanced analytics are required to improve the reliability and repeatability of additive manufacturing, and those analytics require data. Large volumes of multimodal data are generated and used throughout the additive manufacturing lifecycle, from material design to part design and simulation, part printing to post-processing and inspection. To capture and link that diverse Big Data together, we have designed and developed a federated multimodal Big Data storage and analytics platform comprised of three tiers-a distributed polyglot data storage and analysis tier with different repositories for different data structures, a metadata knowledge graph tier for modeling the data and their relationships across the various repositories, and a user interface tier for visualizing, exploring and invoking analytics on the data. The platform has been used to integrate a collection of previously disparate steps to optimize the process parameters used to build a new additive material, enabling materials scientists and other non-software experts to capture, visualize and analyze the requisite data through a single user interface.

The recent successes of commercial cognitive and AI applications have cast a spotlight on knowledge graphs and the benefits of consuming structured semantic data. Today, knowledge graphs are ubiquitous to the extent that organizations often view them as a “single source of truth” for all of their data and other digital artifacts. In most organizations, however, Big Data comes in many different forms including time series, images, and unstructured text, which often are not suitable for efficient storage within a knowledge graph. This paper presents the Semantics Toolkit (SemTK), a framework that enables access to polyglot-persistent Big Data stores while giving the appearance that all data is fully captured within a knowledge graph. SemTK allows data to be stored across multiple storage platforms (e.g., Big Data stores such as Hadoop, graph databases, and semantic triple stores) - with the best-suited platform adopted for each data type - while maintaining a single logical interface and point of access, thereby giving users a knowledge-driven veneer across their data. We describe the ease of use and benefits of constructing and querying polystore knowledge graphs with SemTK via four industrial use cases at GE.

The era of precision medicine is best exemplified by the growing reliance on next-generation sequencing (NGS) technologies to provide improved disease diagnosis and targeted therapeutic selection. Well-established NGS data analysis software tools, in their unmodified form, can take days to identify and interpret single nucleotide and structural variations in DNA for a single patient. To improve sample analysis throughput, we developed a highly parallel end-to-end next-generation DNA sequencing data analysis pipeline in Hadoop. In our pipeline, each step is parallelized not only across samples but also within each individual sample, achieving a 30× speedup over a single server workflow execution. Furthermore, we extensively evaluate the viability of having our Hadoop-based pipeline as part of a larger commercial genomic services offering-we demonstrate how our pipeline scales sub-linearly both with the number of samples being analyzed and with the depth of coverage of those samples. In particular, on our commodity cluster, 10× as many samples resulted in only a 2.24× increase in the execution time, and a 4× increase in coverage depth resulted in only a 2.53× growth in execution time. We anticipate that such improvements will allow large cohort populations to be analyzed in parallel, and can fundamentally change the way DNA sequencing analyses are used by both researchers and clinicians.

Hadoop and the associated MapReduce paradigm, has become the de facto platform for cost-effective analytics over “Big Data”. There is an increasing number of MapReduce applications associated with live business intelligence that require completion time guarantees. In this work, we introduce and analyze a set of complementary mechanisms that enhance workload management decisions for processing MapReduce jobs with deadlines. The three mechanisms we consider are the following: 1) a policy for job ordering in the processing queue; 2) a mechanism for allocating a tailored number of map and reduce slots to each job with a completion time requirement; 3) a mechanism for allocating and deallocating (if necessary) spare resources in the system among the active jobs. We analyze the functionality and performance benefits of each mechanism via an extensive set of simulations over diverse workload sets. The proposed mechanisms form the integral pieces in the performance puzzle of automated workload management in MapReduce environments.

Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not a ect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.

Spatial object association, also referred to as crossmatch of spatial datasets, is the problem of identifying and comparing objects in two or more datasets based on their positions in a common spatial coordinate system. In this work, we evaluate two crossmatch algorithms that are used for astronomical sky surveys, on the following database system architecture configurations: (1) Netezza Performance Serverreg, a parallel database system with active disk style processing capabilities, (2) MySQL Cluster, a high-throughput network database system, and (3) a hybrid configuration consisting of a collection of independent database system instances with data replication support. Our evaluation provides insights about how architectural characteristics of these systems affect the performance of the spatial crossmatch algorithms. We conducted our study using real use-case scenarios borrowed from a large-scale astronomy application known as the large synoptic survey telescope (LSST).

This paper describes our experience to date employing the systematic mapping and optimization of large- scale scientific application workflows to current and future parallel platforms. The overall goal of the project is to integrate a set of system layers - application program, compiler, run-time environment, knowledge representation, optimization framework, and workflow manager - and through a systematic strategy for workflow mapping, our approach will exploit the vast machine resources available in such parallel platforms to dramatically increase the productivity of application programmers. In this paper, we describe the representation of a biomedical imaging application as a workflow, our early experiences in integrating the set of tools brought together for this project, and implications for future applications.

This paper presents the application of a component-based Grid middleware system for processing extremely large images obtained from digital microscopy devices. We have developed parallel, out-of-core techniques for different classes of data processing operations employed on images from confocal microscopy scanners. These techniques are combined into a data preprocessing and analysis pipeline using the component-based middleware system. The experimental results show that: 1) our implementation achieves good performance and can handle very large datasets on high-performance Grid nodes, consisting of computation and/or storage clusters and 2) it can take advantage of Grid nodes connected over high-bandwidth wide-area networks by combining task and data parallelism.

In many data analysis applications, application-level parameters influence the execution time of the data analysis method or program. Some of these parameters also affect the accuracy of output of the analysis. In this work, we investigate execution strategies for adaptive data analysis applications where the user is willing to trade-off accuracy of output for performance gain and vice-versa. In order to meet the user defined quality of service requirements, the system must dynamically select values for the parameters during execution. We propose algorithms for adaptive processing of image tiles at different resolutions so that user defined requirements in terms of accuracy of the result and execution time constraints can be satisfied. We develop heuristics for estimation of accuracy vs performance characteristics of image tiles and for scheduling of the tiles for processing. We implement a demand-driven strategy for parallel execution of these heuristics on a parallel machine. We evaluate our approach for analysis of large images from digitized microscopy scanners.

This paper describes a new project that systematically addresses the enormous complexity of mapping applications to current and future parallel platforms. By integrating the system layers - domain-specific environment, application program, compiler, run-time environment, performance models and simulation, and workflow manager - and through a systematic strategy for application mapping, our approach exploit the vast machine resources available in such parallel platforms to dramatically increase the productivity of application programmers. This project brings together computer scientists in the areas represented by the system layers (i.e., language extensions, compilers, run-time systems, workflows) together with expertise in knowledge representation and machine learning. With expert domain scientists in molecular dynamics (MD) simulation, we are developing our approach in the context of a specific application class which already targets environments consisting of several hundreds of processors. In this way, we gain valuable insight into a generalizable strategy, while simultaneously producing performance benefits for existing and important applications.

This paper is concerned with efficient execution of a pipeline of data processing operations on very large images obtained from confocal microscopy instruments. We describe parallel, out-of-core algorithms for each operation in this pipeline. One of the challenging steps in the pipeline is the warping operation using inverse mapping based methods. We propose and investigate a set of algorithms to handle the warping computations on storage clusters. Our experimental results show that the proposed approaches are scalable both in terms of number of processors and the size of images.

We present a combined task- and data-parallel approach for distributed execution of pre-processing operations to support efficient evaluation of polygonal aggregation queries on digitized microscopy images. Our approach targets out-of-core, pipelined processing of very large images on active storage clusters. Our experimental results show that the proposed approach is scalable both in terms of number of processors and the size of images