Research process automation–the reliable, efficient, and reproducible execution of linked sets of actions on scientific instruments, computers, data stores, and other resources–has emerged as an essential element of modern science. We report here on new services within the Globus research data management platform that enable the specification of diverse research processes as reusable sets of actions, flows , and the execution of such flows in heterogeneous research environments. To support flows with broad spatial extent (e.g., from scientific instrument to remote data center) and temporal extent (from seconds to weeks), these Globus automation services feature: (1) cloud hosting for reliable execution of even long-lived flows despite sporadic failures; (2) a simple specification and extensible asynchronous action provider API, for defining and executing a wide variety of actions and flows involving heterogeneous resources ; (3) an event-driven execution model for automating execution of flows in response to arbitrary events; and (4) a rich security model enabling authorization delegation mechanisms for secure execution of long-running actions across distributed resources . These services permit researchers to outsource and automate the management of a broad range of research tasks to a reliable, scalable, and secure cloud platform. We present use cases for Globus automation services, describe their design and implementation, present microbenchmark studies, and review experiences applying the services in a range of applications.

ƒuncX is a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. Unlike centralized FaaS systems, ƒuncX decouples the cloud-hosted management functionality from the edge-hosted execution functionality. ƒuncX's endpoint software can be deployed, by users or administrators, on arbitrary laptops, clouds, clusters, and supercomputers, in effect turning them into function serving systems. ƒuncX's cloud-hosted service provides a single location for registering, sharing, and managing both functions and endpoints. It allows for transparent, secure, and reliable function execution across the federated ecosystem of endpoints—enabling users to route functions to endpoints based on specific needs. ƒuncX uses containers (e.g., Docker, Singularity, and Shifter) to provide common execution environments across endpoints. ƒuncX implements various container management strategies to execute functions with high performance and efficiency on diverse ƒuncX endpoints. ƒuncX also integrates with an in-memory data store and Globus for managing data that may span endpoints. We motivate the need for ƒuncX, present our prototype design and implementation, and demonstrate, via experiments on two supercomputers, that ƒuncX can scale to more than 130000 concurrent workers. We show that ƒuncX's container warming-aware routing algorithm can reduce the completion time for 3,000 functions by up to 61% compared to a randomized algorithm and the in-memory data store can speed up data transfers by up to 3x compared to a shared file system.

The function as a service paradigm aims to abstract the complexities of managing computing infrastructure for users. While adoption in industry has been swift, we have yet to see widespread adoption in academia. This is in part due to barriers such as the need to access large research data, diverse hardware requirements, monolithic code bases, and existing systems available to researchers. We describe funcX, a federated functionas-a-service platform that addresses important requirements for use of FaaS in research computing. We outline how funcX has been used in early science deployments.

Collaborative e-Science applications often need to manage large numbers of user identities, profiles, and groups. However, developing and maintaining such capabilities is often challenging given the plethora of security protocols available and requirements for scalable, robust, and highly available implementations. Globus Nexus is a professionally hosted Platform-as-a-Service that provides these capabilities for collaborative e-Science applications, with a particular focus on the needs of scientific communities. It provides features such as identity provisioning, identity federation, profile management, user-oriented group management, and branded web interfaces that are important to many e-Science applications. Globus Nexus implements best practices approaches for each of these features for example using delegated security protocols such as OAuth, provides sophisticated workflows for actions such as email validation, and implements complex user-defined policies regarding permissible actions. We present here Globus Nexus' capabilities, motivate design choices, and present results that characterize the scalability, reliability, and availability of its implementation and deployment.

Globus Nexus is a flexible and powerful Platform-as-a-Service to which developers can outsource identity, group, and profile management needs. By providing these frequently important but always challenging capabilities as a service, accessible over the network, Globus Nexus streamlines web application development and makes it easy for individuals, teams, and institutions to create collaborative web applications such as science gateways for the science community. We introduce the capabilities of this platform and review representative applications.