With cloud providers constantly seeking the best infrastructure trade-off between performance delivered to customers and overall energy/utilization efficiency of their data-centres, hardware disaggregation comes in as a new paradigm for dynamically adapting the data-centre infrastructure to the characteristics of the running workloads. Such an adaptation enables an unprecedented level of efficiency both from the standpoint of energy and the utilization of system resources. In this paper, we present - ThymesisFlow - the first, to our knowledge, full-stack prototype of the holy-grail of disaggregation of compute resources: pooling of remote system memory. Thymesis-Flow implements a HW/SW co-designed memory disaggregation interconnect on top of the POWER9 architecture, by directly interfacing the memory bus via the OpenCAPI port. We use ThymesisFlow to evaluate how disaggregated memory impacts a set of cloud workloads, and we show that for many of them the performance degradation is negligible. For those cases that are severely impacted, we offer insights on the underlying causes and viable cross-stack mitigation paths.

Current datacenters are based on server machines, whose mainboard and hardware components form the baseline, monolithic building block that the rest of the system software, middleware and application stack are built upon. This leads to the following limitations: (a) resource proportionality of a multi-tray system is bounded by the basic building block (mainboard), (b) resource allocation to processes or virtual machines (VMs) is bounded by the available resources within the boundary of the mainboard, leading to spare resource fragmentation and inefficiencies, and (c) upgrades must be applied to each and every server even when only a specific component needs to be upgraded. The dRedBox project (Disaggregated Recursive Datacentre-in-a-Box) addresses the above limitations, and proposes the next generation, low-power, across form-factor datacenters, departing from the paradigm of the mainboard-as-a-unit and enabling the creation of function-block-as-a-unit. Hardware-level disaggregation and software-defined wiring of resources is supported by a full-fledged Type-1 hypervisor that can execute commodity virtual machines, which communicate over a low-latency and high-throughput software-defined optical network. To evaluate its novel approach, dRedBox will demonstrate application execution in the domains of network functions virtualization, infrastructure analytics, and real-time video surveillance.

Indoor Positioning Systems (IPS) are gaining market momentum, mainly due to the significant reduction of sensor cost (on smartphones or standalone) and leveraging standardization of related technology. Among various alternatives for accurate and cost-effective IPS, positioning based on the Magnetic Field has proven popular, as it does not require specialized infrastructure. Related experimental results have demonstrated good positioning accuracy. However, when transitioned to production deployments, these systems exhibit serious drawbacks to make them practical: a) accuracy fluctuates significantly across smartphone models and configurations and b) costly continuous manual fingerprinting of the area is required. In this paper we propose Copernicus, a self-learning, adaptive system that is shown to exhibit improved accuracy across different smartphone models. Copernicus leverages a minimal deployment of Bluetooth Low Energy (BLE) Beacons to infer the trips of users, learn and eventually build tailored Magnetic Maps for every smartphone model for the specific indoor area. Our experimental results show the positive impact in the positioning, even in case of minimal learning.

Task-based programming models for shared memory—such as Cilk Plus and OpenMP 3—are well established and documented. However, with the increase in parallel, many-core, and heterogeneous systems, a number of research-driven projects have developed more diversified task-based support, employing various programming and runtime features. Unfortunately, despite the fact that dozens of different task-based systems exist today and are actively used for parallel and high-performance computing (HPC), no comprehensive overview or classification of task-based technologies for HPC exists. In this paper, we provide an initial task-focused taxonomy for HPC technologies, which covers both programming interfaces and runtime mechanisms. We demonstrate the usefulness of our taxonomy by classifying state-of-the-art task-based environments in use today.

The latest tremendous growth in the Internet traffic has determined the entry into a new era of mega-data centers meant to deal with this explosion of data traffic. However, this big data with its dynamically changing traffic patterns and flows might result in degradations of the application performance eventually affecting the network operators' revenue. In this context, there is a need for an intelligent and efficient network management system that makes the best use of the available bisection bandwidth abundance to achieve high utilization and performance. This paper proposes OFLoad, an OpenFlow-based dynamic load balancing strategy for data center networks that enables the efficient use of the network resources capacity. A real experimental prototype is built and the proposed solution is compared against other solutions from the literature in terms of load-balancing. The aim of OFLoad is to enable the instant configuration of the network by making the best use of the available resources at the lowest cost and complexity.

Big data analytics is an indispensable tool in transforming science, engineering, medicine, health-care, finance and ultimately business itself. With the explosion of data sizes and need for shorter time-to-solution, in-memory platforms such as Apache Spark gain increasing popularity. In this context, data shuffling, a particularly difficult transformation pattern, introduces important challenges. Specifically, data shuffling is a key component of complex computations that has a major impact on the overall performance and scalability. Thus, speeding up data shuffling is a critical goal. To this end, state-of-the-art solutions often rely on overlapping the data transfers with the shuffling phase. However, they employ simple mechanisms to decide how much data and where to fetch it from, which leads to sub-optimal performance and excessive auxiliary memory utilization for the purpose of prefetching. The latter aspect is a growing concern, given evidence that memory per computation unit is continuously decreasing while interconnect bandwidth is increasing. This paper contributes a novel shuffle data transfer strategy that addresses the two aforementioned dimensions by dynamically adapting the prefetching to the computation. We implemented this novel strategy in Spark, a popular inmemory data analytics framework. To demonstrate the benefits of our proposal, we run extensive experiments on an HPC cluster with large core count per node. Compared with the default Spark shuffle strategy, our proposal shows: up to 40 percent better performance with 50 percent less memory utilization for buffering and excellent weak scalability.

Big data analytics is an indispensable tool in transforming science, engineering, medicine, healthcare, finance and ultimately business itself. With the explosion of data sizes and need for shorter time-to-solution, in-memory platforms such as Apache Spark gain increasing popularity. However, this introduces important challenges, among which data shuffling is particularly difficult: on one hand it is a key part of the computation that has a major impact on the overall performance and scalability so its efficiency is paramount, while on the other hand it needs to operate with scarce memory in order to leave as much memory available for data caching. In this context, efficient scheduling of data transfers such that it addresses both dimensions of the problem simultaneously is non-trivial. State-of-the-art solutions often rely on simple approaches that yield sub optimal performance and resource usage. This paper contributes a novel shuffle data transfer strategy that dynamically adapts to the computation with minimal memory utilization, which we briefly underline as a series of design principles.

Future supercomputers will need to support both traditional HPC applications and Big Data/High Performance Analysis applications seamlessly in a common environment. This motivates traditional job scheduling systems to support malleable jobs along with allocations that can dynamically change in size, in order to adapt the amount of resources to the actual current need of the different applications. It also calls for future innovative HPC applications to adapt to this environment, and provide some level of malleability for releasing underutilized resources to other tasks. In this paper, we present and compare two different methodologies to support such malleable MPI applications: 1)using checkpoint/restart and the SCR library, and 2) using dynamic data redistribution and the ULFM API and runtime. We examine their effects on application execution times as well as their impact on resource management.

As data volumes and the need for timely analysis grow, Big Data analytics frameworks have to scale out to hundred or even thousands of commodity servers. While such a scale-out is crucial to sustain desired computational throughput/latency and storage capacity, it comes at the cost of increased network traffic volumes and multiplicity of traffic patterns. Despite the sheer reality of the dependency between datacenter network (DCN) and time-to-insight through big data analysis, our experience as active networking researchers conveys that a large fraction of DCN research experimentation is conducted on network traces and/or synthetic flow traces. And while the respective results are often valuable as standalone contributions, in practice it turns out extremely difficult to quantitatively assess how the reported network optimization results translate to performance or fault-tolerance improvement for actual analytics runtimes, e.g., due to the ability of these runtimes to overlap communication with computation. This paper presents MRemu, an emulation-based framework for conducting reproducible datacenter network research using accurate MapReduce workloads and at system scales that are relevant to the size of target deployments, albeit without requiring access to a hardware infrastructure of such scale. We choose the MapReduce (MR) framework as a design point, for it is a common representative of the most widely deployed frameworks for analysis of large volumes of - structured and unstructured - data and is reported to be highly sensitive to network performance. With MRemu, it is possible to quantify the impact of various network design parameters and software-defined control techniques to key performance indicators of a given MR application. We show through targeted experimental validation that MRemu exhibits high fidelity, when compared to the performance of MR applications on a real scale-out cluster of 16 high-end servers.

The rise of Internet of Things sensors, social networking and mobile devices has led to an explosion of available data. Gaining insights into this data has led to the area of Big Data analytics. The MapReduce framework, as implemented in Hadoop, is one of the most popular frameworks for Big Data analysis. To handle the ever-increasing data size, Hadoop is a scalable framework that allows dedicated, seemingly unbound numbers of servers to participate in the analytics process. Response time of an analytics request is an important factor for time to value/insights. While the compute and disk I/O requirements can be scaled with the number of servers, scaling the system leads to increased network traffic. Arguably, the communication-heavy phase of MapReduce contributes significantly to the overall response time, the problem is further aggravated, if communication patterns are heavily skewed, as is not uncommon in many MapReduce workloads. In this paper we present a system that reduces the skew impact by transparently predicting data communication volume at runtime and mapping the many end-to-end flows among the various processes to the underlying network, using emerging software-defined networking technologies to avoid hotspots in the network. Dependent on the network oversubscription ratio, we demonstrate reduction in job completion time between 3% and 46% for popular MapReduce benchmarks like Sort and Nutch.

SUMMARY We consider a hybrid electronic packet switched and optical circuit switched interconnection network for future high performance computing and datacenter systems. Given the logical task‐to‐task communication graph of an application, our objective is to cluster the logical parallel tasks to compute resources and configure the (reconfigurable) optical part of the hybrid interconnect to efficiently serve application communication requirements. We formulate the clustering and topology configuration problem in such a network, prove that it is NP‐complete, and provide an optimal algorithm to solve it based on an integer linear programming formulation. The integer linear programming algorithm is used to optimally solve small‐scale instances of the problem for the purpose of obtaining performance bounds. Aiming at large‐scale, we also present a heuristic based on simulated annealing that trades‐off performance for responsiveness. We measure the performance of a hybrid interconnect employing the proposed algorithm using real workloads, as well as extrapolated traffic, and compare it against application mapping on conventional fixed, electronic‐only interconnects based on toroidal topologies. Copyright © 2013 John Wiley & Sons, Ltd.

The recent trend for Software as a Service and other types of cloud services is driving demand for data centers of ever increasing scale. This will require the scaling up and scaling out of existing data center architectures and will ultimately need new architectures featuring new topologies for the data center networks that interconnect its constituent nodes. The preferred tool for the detailed evaluation of such designs is simulation, as it gives finer detail than analysis at lower cost than test bed evaluation. Realistic simulation times require that the simulation itself be capable of being scaled out, i.e., it should be amenable to parallelization. We describe a simulation framework, together with a methodology for partitioning the relevant simulation models to allow their parallel implementation, and demonstrate its validity by applying it to the simulation of a hybrid optical/electrical network architecture using a cluster of high-end servers. We report on results capturing the performance of our simulator and discuss how these are associated with the underlying hardware hosting the simulation.

Stream computing is an emerging computational model for performing complex operations on and across multi-source, high volume data flows. Given that the deployment of the model has only started, the pool of mature applications employing this model is fairly small, and therefore the availability of workloads for various types of applications is scarce. Thus, there is a need for synthetic generation of large-scale workloads for evaluation of stream computing applications at scale. This paper presents a framework for producing synthetic workloads for stream computing systems. Our framework extends known random graph generation concepts with stream computing specific features, providing researchers with realistic input stream graphs and allowing them to focus on system development, optimization and analysis. Serving the goal of covering a disparity of potential applications, the presented framework exhibits high user-controlled configurability. The produced workloads could be used to drive simulations for performance evaluation and for proof-of-concept prototyping of processing, networking and operating system hardware and software.

Although the problem of dimensioning an optical transport network is not new, the consideration of signal quality degradation caused by the optical medium calls for revisiting the problem in the context of dimensioning optical wavelength division multiplexing (WDM) networks. This paper addresses the issue of minimum-cost planning of long-reach WDM networks in combination with optoelectronic signal regeneration as a countermeasure for sanitizing the signal quality of lightpaths that are found to be impaired. The commonly used method of placing regenerators proportionally to the physical distance covered by a lightpath is evaluated in a realistic dimensioning scenario and for various heterogeneity degrees of optical equipment, showing that it is plagued with a serious tradeoff between efficacy and cost of regeneration. As a remedy, we propose a novel method for design/dimensioning and regeneration placement for WDM networks that employs impairment-awareness. Through experimentation with real optical network configurations and for varying heterogeneity of optical equipment, the proposed method is shown to break the aforementioned tradeoff, resulting in significant reduction in regeneration effort compared to distance-based regeneration. This is achieved without compromising the signal quality of any of the lightpaths selected by the dimensioning process and with increased cost efficiency.

This paper focuses on next generation ubiquitous networks supporting the Future Internet. In this context, it proposes an architecture and an integration framework of wireless and wired network technologies supporting a variety of services with differing service requirements. More specifically the i

In spite of significant work on real-time content distribution, providing consistent service quality across the Internet remains a challenge. Multicast applications further exacerbate that problem as the number of participants-clients, providers and administrative domains-and the level of heterogeneity in a single session is increased. Recently the increasing complexity of managing and operating the Internet has sparked off interest in autonomous networking. In this paper, we introduce vital enhancements to the elementary single source multicast service that facilitates incremental deployment, self-configuration, self-optimization and self-healing. We further introduce an autonomic multicast architecture where a self-monitoring network layer multicast automatically adapts the network layer forwarding service to meet the service quality requirements requested by application level services.

This paper presents a media- and TCP-friendly rate-based congestion control algorithm (MTFRCC) for scalable video streaming in the Internet. The algorithm integrates two new techniques: i) a utility-based model using the rate-distortion function as the application utility measure for optimizing the overall video quality; and ii) a two-timescale approach of rate averages (long-term and short-term) to satisfy both media and TCP-friendliness. We evaluate our algorithm through simulation and compare the results against the TCP-friendly rate control (TFRC) algorithm. For assessment, we consider five criteria: TCP fairness, responsiveness, aggressiveness, overall video quality, and smoothness of the resulting bit rate. Our simulation results manifest that MTFRCC performs better than TFRC for various congestion levels, including an improvement of the overall video quality.

Content distribution is becoming increasingly important with the decreasing cost of broadband access and the growing number of Internet users. Many of today's content distribution applications require a group communication infrastructure. Source-Specific Multicast (SSM) constitutes a promising alternative to the slow takeoff of multicast as a ubiquitous service, whether at the network- or application layer. Still, implementing any-source functionality on top of SSM remains an interesting research topic.In this paper we present how application semantics can be applied to multi-source distribution over source-specific trees. Specifically, we present two novel tree management methods that incorporate application semantics and evaluate them through simulations against the state of the art. The evaluation results manifest that our methods significantly improve perceived quality of service in the multi-source sessions we consider.