Modern scientific instruments, such as detectors at synchrotron light sources, generate data at such high rates that online processing is needed for data reduction, feature detection, experiment steering, and other purposes. The same high data rates also demand memory-to-memory streaming from instrument to remote computer, because local computational capacity is limited and data transmissions that engage the file system introduce unacceptable latencies. But efficient and secure memory-to-memory data streaming is challenging to realize in practice, because of a lack of direct external network connectivity for scientific instruments and because of authentication and security requirements. In response, we propose here SciStream, a middlebox-based architecture with control protocols to enable efficient and secure memory-to-memory data streaming between producers and consumers that lack direct network connectivity. We describe the protocols that SciStream uses to establish authenticated and transparent connections between producers and consumers, and we discuss the experiments that we have conducted to evaluate alternative implementation approaches for key SciStream components. Experiments on the Chameleon cloud show that SciStream improves the throughput of a streaming pipeline by an order of magnitude compared with state-of-the-art data transfer methods and adds only ~4μsec latency compared with an ideal scenario in which producers and consumers have direct external connectivity.

With the emergence of programmable network devices that match the performance of fixed function devices, several recent projects have explored in-network computing, where the processing that is traditionally done outside the network is offloaded to the network devices. In-network computing has typically been applied to network functions (e.g., load balancing, NAT, and DNS), caching, data reduction/aggregation, and coordination/consensus functions. In some cases it has been used to accelerate stream-processing tasks that involve small payloads and simple operations. In this work we focus on leveraging in-network computing for stream processing of scientific datasets with large payloads that require complex operations such as floating-point computations and logarithmic functions. We demonstrate in-network computing for a real-world scientific application performing streaming normalization of a 2-D image from a light source experiment. We discuss the challenges we encountered and potential approaches to address them.

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments’ memory directly to a supercomputer’s memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

Data transfer over wide area networks is an integral part of many science workflows that must, for example, move data from scientific facilities to remote resources for analysis, sharing, and storage. Yet despite continued enhancements in data transfer infrastructure (DTI), our previous analyses of approximately 40 billion GridFTP command logs collected over four years from the Globus transfer service show that data transfer nodes (DTNs) are idle (i.e., are performing no transfers) 94.3% of the time. On the other hand, we have also observed periods in which CPU resource scarcity negatively impacts DTN throughput. Motivated by the opportunity to optimize DTI performance, we present here an elastic DTI architecture in which the pool of nodes allocated to DTN activities expands and shrinks over time, based on demand. Our results show that this elastic DTI can save up to ~95% of resources compared with a typical static DTN deployment, with the median slowdown incurred remaining close to one for most of the evaluated scenarios.

Many science workflows are distributed in nature and rely on wide area networks (WANs) to move data between geographically distributed resources for analysis, sharing, and storage. In spite of continued enhancements in campus cyberinfrastructure, data transfer nodes (DTNs) are grossly underutilized. Our previous analysis of logs from 1,800 DTNs shows that they were completely idle for 94.3% of the time in 2017. Motivated by the opportunity to optimize DTN usage, here we present an elastic data transfer infrastructure (DTI) architecture in which the pool of nodes allocated to DTN activities expands and shrinks over time, based on demand. Our results show that this elastic DTI can save up to $\sim 95$% of resources compared with a typical static DTN deployment.

The provisioning of network connections for data transfers that provide quality of service (QoS) over research and education (R&E) networks is currently performed by network operators. For network connections that span multiple administrative domains, network operators have to reach agreements on reservation requirements. As a result, a network reservation request may take from days to weeks to be provisioned. To improve provisioning times and the success rate of multidomain network reservations, we designed and implemented an application-aware orchestration framework for multidomain R&E networks. This framework leverages latest developments in software-defined networking to automate network provisioning in order to democratize access to network reservation through novel APIs. We present the design, implementation, and evaluation of our application-aware orchestration framework. We evaluate our system using Mininet and demonstrate that it provisions 49% more reservations than current state-of-the-art systems within seconds of receiving a request.

Wireless mesh networks (WMNs) are next generation networks that have emerged as a promising technology capable of providing broadband connectivity at low cost, and offers advantages in scenarios where cabling is too expensive or prohibitive. Rural communities are a good example of these kind of scenarios. Deployment of WMNs in rural communities can be done with a low budget, and expansion of the network can be done as the community grows. In this paper we evaluate the performance of a small scale wireless mesh network, comparing two well known routing protocols for WMNs, B.A.T.M.A.N. Advanced and the 802.11s standard, against an OpenFlow controller. Then, by deploying a testbed we verify their bandwidth, datagram loss and jitter. Finally, our results show that both routing protocols have advantages and disadvantages. Besides, the OpenFlow approach shows an acceptable performance compared to the routing protocols; however, some adjustments are needed to optimize the performance of OpenFlow in WMNs. Our contribution in this paper is the characterization of the multimedia service capacity of wireless mesh networks for rural communities.