Large volumes of datasets are continuously generated across various scientific domains. After generation, the big datasets often need to be quickly transferred geographically using bandwidth reservation service provided by dedicated high-performance networks (HPNs) for collaborative data storage and analysis. While the primary requirement of the data transfer is the earliest completion time (ECT), users may also have additional requirements, such as minimizing data transfer costs financially. Striking the right balance between these diverse data transfer needs can be challenging. In this paper, we delve into the trade-off between completion time and cost of data transfers using bandwidth reservation on fixed-bandwidth variable paths in dedicated HPNs. We focus on two types of bandwidth reservation requests (BRRs) and their scheduling: (i) achieving the minimum data transfer cost under the constraint of data transfer deadline, and (ii) achieving the data transfer ECT under the constraint of maximum data transfer cost. To optimize the scheduling of both types of BRRs, we propose two optimal and novel algorithms and then conduct extensive simulations to demonstrate the effectiveness and efficiency of the proposed algorithms.

A wide spectrum of large-scale applications in science, industry, and business domains generate colossal amounts of data that need to be moved across geographical locations for various purposes. Such big data transfer is increasingly carried out over connections with guaranteed bandwidth provisioned in high-performance networks, as exemplified by ESnet via advance bandwidth reservation agents such as OSCARS. Accurate performance prediction of big data transfer is vital for resource management to reserve appropriate bandwidth and optimize resource utilization. Data-driven methods using machine learning offer a promising solution to such prediction capabilities. However, most of these methods highly rely on the quality of training data and might suffer from low prediction accuracy when performance measurements are collected in environments with high dynamics or under latent effects. We propose a performance prediction method to help end users obtain a good estimate of achievable throughput, which could also be used by network managers to infer expected bandwidth usage and hence optimize bandwidth reservation and utilization. Based on Gaussian Process Regression (GPR), our approach is able to automatically detect intrinsic noise and identify latent effects to enhance prediction accuracy. We implement and evaluate the proposed method using performance measurements collected in real-life networks and the results demonstrate its effectiveness.

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.

Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-tomemory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0-366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincarέmaps and Lyapunov exponents. For diskto-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.

The network infrastructures have been rapidly upgraded in many high-performance networks (HPNs). However, such infrastructure investment has not led to corresponding performance improvement in big data transfer, especially at the application layer, largely due to the complexity of optimizing transport control on end hosts. We design and implement ProbData, a PRofiling Optimization Based DAta Transfer Advisor, to help users determine the most effective data transfer method with the most appropriate control parameter values to achieve the best data transfer performance. ProbData employs a profiling optimization-based approach to exploit the optimal operational zone of various data transfer methods in support of big data transfer in extreme-scale scientific applications. We present a theoretical framework of the optimized profiling approach employed in ProbData as well as its detailed design and implementation. The advising procedure and performance benefits of ProbData are illustrated and evaluated by proof-of-concept experiments in real-life networks.

High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.

The transfer of big data is increasingly supported by dedicated channels in high-performance networks, where transport protocols play an important role in maximizing application-level throughput and link utilization. The performance of transport protocols largely depend on their control parameter settings, but it is prohibitively time consuming to conduct an exhaustive search in a large parameter space to find the best set of parameter values. We propose FastProf, a stochastic approximation-based transport profiler, to quickly determine the optimal operational zone of a given data transfer protocol/method over dedicated channels. We implement and test the proposed method using both emulations based on real-life performance measurements and experiments over physical connections with short (2ms) and long (380ms) delays. Both the emulation and experimental results show that FastProf significantly reduces the profiling overhead while achieving a comparable level of end-to-end throughput performance with the exhaustive search-based approach.

Wide-area data transfers in high-performance computing and big data scenarios are increasingly being carried over dedicated network connections that provide high capacities at low loss rates. UDP-based transport protocols are expected to be particularly well-suited for such transfers but their performance is relatively unexplored over a wide range of connection lengths, compared to TCP over shared connections. We present extensive throughput measurements of UDP-based Data Transfer (UDT) over a suite of physical and emulated 10 Gbps connections. In sharp contrast to current UDT analytical models, these measurements indicate much more complex throughput dynamics that are sensitive to the connection modality, protocol parameters, and round-trip times. Lyapunov exponents estimated from the Poincaré maps of UDT traces clearly indicate regions of instability and complex dynamics. We propose a simple model based on the ramp-up and sustainment regimes of a generic transport protocol, which qualitatively illustrates the dominant monotonicity and concavity properties of throughput profiles and relates them to Lyapunov exponents. These measurements and analytical results together enable us to comprehensively evaluate UDT performance and select parameters to achieve high throughput, and they also provide guidelines for designing effective transport protocols for dedicated connections.

We propose and develop an integrated transport solution to big data movement in high-performance networks in support of data-and network-intensive applications in various science domains. This solution integrates three major components, i.e. (i) transport-support workflow optimization, (ii) transport profile generation, and (iii) transport protocol design, into a unified framework.

An increasing number of high-performance networks provision dedicated channels through circuit-switching or MPLS/GMPLS tunneling techniques to support large data transfer. The link bandwidths of these networks are typically shared by multiple users through advance scheduling and reservation. The sheer volume of data transfer across such networks in a national or international scope requires a significant amount of energy on a daily basis. However, most existing bandwidth scheduling algorithms only concern traditional objectives such as data transfer time minimization, and very limited efforts have been devoted to energy efficiency in high-performance networks. In this paper, we adopt a practical power model and formulate an advance instant bandwidth scheduling problem to minimize energy consumption under a data transfer deadline constraint. We design a polynomial-time optimal solution to this problem and provide a rigorous correctness proof. The performance superiority of the proposed solution in terms of energy saving is illustrated by extensive results based on both simulated and real-life networks in comparison with existing methods.

Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

High-performance networking technologies and services are being rapidly developed and deployed across the nation and around the globe to support the transfer of large data sets generated by next-generation scientific applications for collaborative data processing, analysis, and storage. However, these networking technologies and services have not been fully utilized mainly because their use often requires considerable domain knowledge and many application users are even not aware of their existence. The main goal of our work is to provide end users an integrated solution to discovering system and network resources and composing end-to-end paths for large data transfer. By leveraging the resource discovery capability previously developed in Network-Aware Data Movement Advisor (NADMA), we propose novel profiling and modeling approaches to characterize various types of resources that are available in end systems, edge segments, and backbone networks, taking into consideration a comprehensive set of performance metrics and network parameters in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we formulate a class of transport-support workflow optimization problems where an appropriate set of technologies and services are selected to compose the best transport-support workflow to meet the user's data transfer request in terms of various performance requirements. We conduct wide-area network experiments to validate the cost models and illustrate the efficacy of the proposed workflow-based transport solution.