Bowtie 2 is a popular short sequence aligner used by many bioinformatics groups, often as part of the Qiita microbial study management platform. One of the key computational steps during alignment is the extraction of seeds from the read sequences and aligning them with the help of the FM Index. This step was memory access-bound, with access latency as the primary limitation due to a pseudo-random-access pattern. In this paper we present the algorithmic changes used and the performance improvements obtained by re-factoring the code to overlap compute with memory pre-fetching.

Microbiome studies have recently transitioned from experimental designs with a few hundred samples to designs spanning tens of thousands of samples. Modern studies such as the Earth Microbiome Project (EMP) afford the statistics crucial for untangling the many factors that influence microbial community composition. Analyzing those data used to require access to a compute cluster, making it both expensive and inconvenient. We show that recent improvements in both hardware and software now allow to compute key bioinformatics tasks on EMP-sized data in minutes using a gaming-class laptop, enabling much faster and broader microbiome science insights.

The IceCube collaboration relies on GPU compute for many of its needs, including ray tracing simulation and machine learning activities. GPUs are however still a relatively scarce commodity in the scientific resource provider community, so we expanded the available resource pool with GPUs provisioned from the commercial Cloud providers. The provisioned resources were fully integrated into the normal IceCube workload management system through the Open Science Grid (OSG) infrastructure and used CloudBank for budget management. The result was an approximate doubling of GPU wall hours used by IceCube over a period of 2 weeks, adding over 3.1 fp32 EFLOP hours for a price tag of about $58k. This paper describes the setup used and the operational experience.

HTCondor is a major workload management system used in distributed high throughput computing (dHTC) environments, e.g., the Open Science Grid. One of the distinguishing features of HTCondor is the native support for data movement, allowing it to operate without a shared filesystem. Coupling data handling and compute scheduling is both convenient for users and allows for significant infrastructure flexibility but does introduce some limitations. The default HTCondor data transfer mechanism routes both the input and output data through the submission node, making it a potential bottleneck. In this document we show that by using a node equipped with a 100 Gbps network interface (NIC) HTCondor can serve data at up to 90 Gbps, which is sufficient for most current use cases, as it would saturate the border network links of most research universities at the time of writing.

Even with Grid technologies, the main mode of access for the current High Performance Computing and High Throughput Computing infrastructures today is logging in via ssh. This mode of access locks scientists to particular machines as it is difficult to move the codes and environments between hosts. In this paper we show how switching the resource access mode to a Condor glide in-based overlay can bring together computational resources from multiple cyber infrastructures. This approach provides scientists with a computational infrastructure anchored around the familiar environment of the desktop computer. Additionally, the approach enhances the reliability of applications and workflows by automatically rerouting jobs to functioning infrastructures. Two different science applications were used to demonstrate applicability, one from the field of astronomy and the other one from earth sciences. We demonstrate that a desktop computer is viable as a submit host and central manager for these kind of glide in overlays. However, issues of ease of use and security need to be considered.

The improvements of the luminosity of the Tevatron Collider require large increases in computing requirements for the CDF experiment which has to be able to increase proportionally the amount of Monte Carlo data it produces. This is, in turn, forcing the CDF collaboration to move beyond the use of dedicated resources and to exploit grid resources. CDF has been running a set of CDF Analysis Farm (CAFs), which are submission portals to dedicated pools, and LcgCAF is basically a reimplementation of the CAF model in order to access grid resources by using the LCG/EGEE middleware components. By mean of LcgCAF CDF users can submit analysis jobs with the same mechanism adopted for the dedicated farms and at the same time the grid resources are accessed without any specific software requirements for the sites. This is obtained using Parrot for the experiment code distribution and Frontier for the run condition database availability on the worker nodes. Currently many sites in Italy and in Europe are accessed through this portal in order to produce Monte Carlo data and in one year of operations we expect about 100,000 grid jobs submitted by the CDF users. We review here the setup used to submit jobs and retrieve the output, including the grid components CDF-specific configuration. The batch and interactive monitor tools developed to allow users to verify the jobs status during their lifetimes in the grid environment are described. We analyze the efficiency and typical failure modes of the current grid infrastructure reporting the performances of different parts of the used system.