MOTIVATION: Developing a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator. RESULTS: We developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science. AVAILABILITY AND IMPLEMENTATION: The PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Workflow managers for scientific analysis provide a high-level programming platform facilitating standardization, automation, collaboration and access to sophisticated computing resources. The Galaxy workflow manager provides a prime example of this type of platform. As compositions of simpler tools, workflows effectively comprise specialized computer programs implementing often very complex analysis procedures. To date, no simple way to automatically test Galaxy workflows and ensure their correctness has appeared in the literature. RESULTS: With wft4galaxy we offer a tool to bring automated testing to Galaxy workflows, making it feasible to bring continuous integration to their development and ensuring that defects are detected promptly. wft4galaxy can be easily installed as a regular Python program or launched directly as a Docker container-the latter reducing installation effort to a minimum. AVAILABILITY AND IMPLEMENTATION: Available at https://github.com/phnmnl/wft4galaxy under the Academic Free License v3.0. CONTACT: [email protected].

The adoption of Big Data technologies can potentially boost the scalability of data-driven biology and health workflows by orders of magnitude. Consider, for instance, that technologies in the Hadoop ecosystem have been successfully used in data-driven industry to scale their processes to levels much larger than any biological-or health-driven work attempted thus far. In this work we demonstrate the scalability of a sequence alignment pipeline based on technologies from the Hadoop ecosystem - namely, Apache Flink and Hadoop MapReduce, both running on the distributed Apache YARN platform. Unlike previous work, our pipeline starts processing directly from the raw BCL data produced by Illumina sequencers. A Flink-based distributed algorithm reconstructs reads from the Illumina BCL data, and then demultiplexes them - analogously to the bcl2fastq2 program provided by Illumina. Subsequently, the BWA-MEM-based distributed aligner from the Seal project is used to perform read mapping on the YARN platform. While the standard programs by Illumina and BWA-MEM are limited to shared-memory parallelism (multi-threading), our solution is completely distributed and can scale across a large number of computing nodes. Results show excellent pipeline scalability, linear in the number of nodes. In addition, this approach automatically benefits from the robustness to hardware failure and transient cluster problems provided by the YARN platform, as well as the scalability of the Hadoop Distributed File System. Moreover, this YARN-based approach complements the up-and-coming version 4 of the GATK toolkit, which is based on Spark and therefore can run on YARN. Together, they can be used to form a scalable complete YARN-based variant calling pipeline for Illumina data, which will be further improved with the arrival of distributed in-memory filesystem technology such as Apache Arrow, thus removing the need to write intermediate data to disk.

In this work we describe pyEHR, a new toolkit for building scalable clinical/phenotypic data management systems for biomedical research applications. The toolkit uses openEHR formalisms to guarantee the decoupling of clinical data descriptions from implementation details, and NoSQL technologies, or next-generation SQL ones, to provide scalable storage back-ends.

UNLABELLED: End-to-end next-generation sequencing microbiology data analysis requires a diversity of tools covering bacterial resequencing, de novo assembly, scaffolding, bacterial RNA-Seq, gene annotation and metagenomics. However, the construction of computational pipelines that use different software packages is difficult owing to a lack of interoperability, reproducibility and transparency. To overcome these limitations we present Orione, a Galaxy-based framework consisting of publicly available research software and specifically designed pipelines to build complex, reproducible workflows for next-generation sequencing microbiology data analysis. Enabling microbiology researchers to conduct their own custom analysis and data manipulation without software installation or programming, Orione provides new opportunities for data-intensive computational analyses in microbiology and metagenomics. AVAILABILITY AND IMPLEMENTATION: Orione is available online at http://orione.crs4.it.

SUMMARY: BioBlend.objects is a new component of the BioBlend package, adding an object-oriented interface for the Galaxy REST-based application programming interface. It improves support for metacomputing on Galaxy entities by providing higher-level functionality and allowing users to more easily create programs to explore, query and create Galaxy datasets and workflows. AVAILABILITY AND IMPLEMENTATION: BioBlend.objects is available online at https://github.com/afgane/bioblend. The new object-oriented API is implemented by the galaxy/objects subpackage.

SUMMARY: Hadoop MapReduce-based approaches have become increasingly popular due to their scalability in processing large sequencing datasets. However, as these methods typically require in-depth expertise in Hadoop and Java, they are still out of reach of many bioinformaticians. To solve this problem, we have created SeqPig, a library and a collection of tools to manipulate, analyze and query sequencing datasets in a scalable and simple manner. SeqPigscripts use the Hadoop-based distributed scripting engine Apache Pig, which automatically parallelizes and distributes data processing tasks. We demonstrate SeqPig's scalability over many computing nodes and illustrate its use with example scripts. AVAILABILITY AND IMPLEMENTATION: Available under the open source MIT license at http://sourceforge.net/projects/seqpig/

SUMMARY: SEAL is a scalable tool for short read pair mapping and duplicate removal. It computes mappings that are consistent with those produced by BWA and removes duplicates according to the same criteria employed by Picard MarkDuplicates. On a 16-node Hadoop cluster, it is capable of processing about 13 GB per hour in map+rmdup mode, while reaching a throughput of 19 GB per hour in mapping-only mode. AVAILABILITY: SEAL is available online at http://biodoop-seal.sourceforge.net/.

High Performance Computational Clusters are, in general, rather rigid objects that present to their user a limited number of degrees of freedom related, usually, only to the specification of the resources requested and to the selection of specific applications and libraries. While in standard production environments this is reasonable and actually desirable, it can become an hindrance when one needs a dynamic and flexible computational environment, for instance for experiments and evaluation, where very different computational approaches, e.g., map-reduce, standard parallel jobs and virtual HPC clusters need to coexist on the same physical facility. In this paper we will present our efforts to address some of these challenges while maintaining a unified cluster management environment.

MapReduce has become increasingly popular as a simple and efficient paradigm for large-scale data processing. One of the main reasons for its popularity is the availability of a production-level open source implementation, Hadoop, written in Java. There is considerable interest, however, in tools that enable Python programmers to access the framework, due to the language's high popularity. Here we present a Python package that provides an API for both the MapReduce and the distributed file system sections of Hadoop, and show its advantages with respect to the other available solutions for Hadoop Python programming, Jython and Hadoop Streaming.

The rapid growth in the throughput to cost ratio of experimental data production technologies is generating vast amounts of scientific data, often organized into "large" objects (genomes, bio-images) exhibiting complex internal structures. Frequently, datasets must be shared between multiple research groups interested not only in the final results, but also in how they are produced. The practical difficulties of moving terabytes or more of data across the network, as well as the need to maintain a clear separation between software stack and storage infrastructure, are thus raising interest in the use of virtual clusters for HPC and data intensive applications. In this paper we employ a MapReduce implementation of an image analysis pipeline used by deep sequencing platforms to analyse different virtual cluster scenarios and their impact on system performance.

Our work focuses on the development of interactive multi-user holographic displays that allow freely moving naked eye participants to share a three dimensional scene with fully continuous, observer independent, parallax. Our approach is based on a scalable design that exploits a specially arranged array of projectors and a holographic screen. The feasibility of such an approach has already been demonstrated with a working hardware and software 7.4M pixel prototype driven at 10-15Hz by two DVI streams. In this short contribution, we illustrate our progress, presenting a 50M pixel display prototype driven by a dedicated cluster hosting multiple consumer level graphic cards.

Bone dissection is an important component of many surgical procedures. In this paper we discuss adaptive techniques for providing real-time haptic and visual feedback during a virtual bone dissection simulation. The simulator is being developed as a component of a training system for temporal bone surgery. We harness the difference in complexity and frequency requirements of the visual and haptic simulations by modeling the system as a collection of loosely coupled concurrent components. The haptic component exploits a multi-resolution representation of the first two moments of the bone characteristic function to rapidly compute contact forces and determine bone erosion. The visual component uses a time-critical particle system evolution method to simulate secondary visual effects, such as bone debris accumulation, blooding, irrigation, and suction.

Bone dissection is an important component of many surgical procedures. In this paper, we discuss a haptic and visual implementation of a bone-cutting burr that is being developed as a component of a training system for temporal bone surgery. We use a physically motivated model to describe the burr-bone interaction, which includes haptic force evaluation, the bone erosion process and the resulting debris. The current implementation, directly operating on a voxel discretization of patient-specific 3D CT and MRI data, is efficient enough to provide real-time feedback on a low-end multiprocessing PC platform.

The aim of the virtual vascular project (ViVa) is to develop tools for the modern hemodynamicist and cardiovascular surgeon to study and interpret the constantly increasing amount of information being produced by noninvasive imaging equipment. In particular, we are developing a system able to process and visualize three-dimensional (3-D) medical data, reconstruct the geometry of arteries of specific patients, and simulate blood flow in them. The initial applications of the system will be for clinical research and training purposes. In a later stage, we will explore the application of the system to surgical planning. ViVa is based on an integrated set of tools, each dedicated to a specific aspect of the data processing and simulation pipeline: image processing and segmentation; real-time 3-D volume visualization; 3-D geometry reconstruction; 3-D mesh generation; and blood flow simulation and visualization.

We describe a new technique for the automatic detection and characterization, in terms of location, size and intensity, of positive hybridization spots on hybridization filter images. We will also discuss how this information can be used to reconstruct the spot deposition pattern on the filter and thus identify the spots. The part of the algorithm dedicated to the detection of spots is derived from a technique used in astronomy for the automatic recognition of galaxies on photographic plates. This technique has been modified so that it becomes able to cope with the strongly varying background typical of hybridization filter images. The technique does not require any human intervention.