Making serverless computing widely applicable requires detailed understanding of performance. Although benchmarking approaches exist, their insights are coarse-grained and typically insufficient for (root cause) analysis of realistic serverless applications, which often consist of asynchronously coordinated functions and services. Addressing this gap, we design and implement ServiTrace, an approach for fine-grained distributed trace analysis and an application-level benchmarking suite for diverse serverless-application architectures. ServiTrace (i) analyzes distributed serverless traces using a novel algorithm and heuristics for extracting a detailed latency breakdown , (ii) leverages a suite of serverless applications representative of production usage, including synchronous and asynchronous serverless applications with external service integrations, and (iii) automates comprehensive, end-to-end experiments to capture application-level performance. Using our ServiTrace reference implementation, we conduct a large-scale empirical performance study in the market-leading AWS environment, collecting over 7.5 million execution traces. We make four main observations enabled by our latency breakdown analysis of median latency, cold starts, and tail latency for different application types and invocation patterns. For example, the median end-to-end latency of serverless applications is often dominated not by function computation but by external service calls, orchestration, and trigger-based coordination; all of which could be hidden without ServiTrace-like benchmarking. We release empirical data under FAIR principles and ServiTrace as a tested, extensible, open-source tool at https://github.com/ServiTrace/ReplicationPackage .

Designing practical exercises to aid in teaching nonfunctional properties of distributed and cloud systems, like scalability, fault tolerance, and performance is hard, as these properties emerge as a result of complex system designs and interactions. In this paper, we describe a hands-on laboratory exercise that can be used as part of a Distributed Systems or Cloud Computing class to help students understand the effect of replication in scalability, fault tolerance and performance of a Big Data platform running on a public cloud provider. Preliminary results indicate that the students like the activity and that working on it improves their understanding of the concepts mentioned above. We have released our laboratory guide on GitHub so that other instructors can use it in their classes: https://github.com/marcelleonlafebre/lab_manualcloud/blob/main/non_functional_demo_lab.md.

We analyze a recently published dataset of 396 real-world cloud architectures deployed on AWS, from companies belonging to a wide range of industries. From this dataset, we identify those architectures that contain HPC or edge components and characterize their designs. Specifically, we investigate the prevalence and interplay of AWS services within these architectures, examine the types of storage systems employed, assess architectural complexity and the use of machine learning services, discuss the implications of our findings and how representative these results are of HPC and edge architectures in the cloud. This characterization provides valuable insights into current industry practices and trends in building robust and scalable HPC and edge solutions in the cloud continuum, and can be valuable for those seeking to better understand how these architectures are being built and to guide new research.

Serverless computing is increasingly used for data-processing applications in both science and business domains. At the core of serverless data-processing systems is the scheduler, which ensures dynamic decisions about task and data placement. Due to the variety of user, cluster, and workload properties, the design space for high-performance and cost-effective scheduling architectures and mechanisms is vast. The large design space is difficult to explore and characterize. To help the system designer disentangle this complexity, we present ExDe, a framework to systematically explore the design space of scheduling architectures and mechanisms. The framework includes a conceptual model and a simulator to assist in design space exploration. We use the framework, and real-world workloads, to characterize the performance of three scheduling architectures and two mechanisms. Our framework is open-source software available on Zenodo.

Data-driven interactive computation is widely used for business analytics, search-based decision-making, and log mining. These applications' short duration and bursty nature makes them a natural fit for serverless computing. Data processing serverless applications are composed of many small tasks. Application tasks that use remote storage encounter bottlenecks in the form of high latency, performance variability, and throttling. Caching has been used to mitigate this bottleneck for intermediate data. However, the use of caching for input data, albeit widely used in industry, has yet to be studied. We present the first performance study of scaling, a key feature of serverless computing, on serverless clusters with input data caches. We compare 8 task placement algorithms and quantify their impact on task slowdown and resource usage before and after scaling. We quantify the consequences of using work stealing. We quantify the performance impact of scaling in the buffer period immediately after scaling. We find up to a 420% increase in task slowdown after scaling without work stealing and a 22% slowdown with work stealing. We also find that cache misses after scaling can lead to an additional 21% resource usage.

Online games with modifiable virtual environments (MVEs) have become highly popular over the past decade. Among them, Minecraft-supporting hundreds of millions of users―is the best-selling game of all time, and is increasingly offered as a service. Although Minecraft is architected as a distributed system, in production it achieves this scale by partitioning small groups of players over isolated game instances. From the approaches that can help other kinds of virtual worlds scale, none is designed to scale MVEs, which pose a unique challenge―a mix between the count and complexity of active in-game constructs, player-created in-game programs, and strict quality of service. Serverless computing emerged recently and focuses, among others, on service scalability. Thus, addressing this challenge, in this work we explore using serverless computing to improve MVE scalability. To this end, we design, prototype, and evaluate experimentally Servo, a serverless backend architecture for MVEs. We implement Servo as a prototype and evaluate it using real-world experiments on two commercial serverless platforms, of Amazon Web Services (AWS) and Microsoft Azure. Results offer strong support that our serverless MVE can significantly increase the number of supported players per instance without performance degradation, in our key experiment by 40 to 140 players per instance, which is a significant improvement over state-of-the-art commercial and open-source alternatives. We release Servo as open-source, on Github: https://github.com/atlarge-research/opencraft.

Multi-task learning is a framework that enforces different tasks to share their knowledge to improve the generalization performance. It is a long-standing active domain that strives to handle several core issues including which tasks are correlated and similar and how to share the knowledge among correlated tasks. Existing works usually do not distinguish the polarity and magnitude of feature weights and commonly rely on linear correlation, due to three major technical challenges in: 1) optimizing the models that regularize feature weight polarity, 2) deciding whether to regularize sign or magnitude, 3) identifying which tasks should share their sign and/or magnitude patterns. To address them, this paper proposes a new multi-task learning framework that can regularize feature weight signs across tasks, beyond the conventional framework for feature weight regularization. We innovatively formulate such sign-regularization problem as a biconvex inequality constrained optimization upon the multiplications among feature weights with slacks. We then propose a new efficient algorithm for the optimization with theoretical guarantees on generalization performance and convergence. Extensive experiments on multiple datasets show the proposed methods’ effectiveness, efficiency, and reasonableness of the regularized feature weighted patterns.

Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.

Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain.

Correctly applying distributed systems concepts is important for software that seeks to be scalable, reliable and fast. For this reason, Distributed Systems is a course included in many Computer Science programs. To both describe current trends in teaching distributed systems and as a reference for educators that seek to improve the quality of their syllabi, we present a review of 51 syllabi of distributed systems courses from top Computer Science programs around the world. We manually curated the syllabi and extracted data that allowed us to identify approaches used in teaching this subject, including choice of topics, book, and paper reading list. We present our results and a discussion on whether what is being taught matches the guidelines of two important curriculum initiatives.

The HotCloudPerf workshop is a meeting venue for academics and practitioners, from experts to trainees, in the field of cloud computing performance. The workshop aims to engage this community, and to lead to the development of new methodological aspects for gaining deeper understanding not only of cloud performance, but also of cloud operation and behavior, through diverse quantitative evaluation tools, including benchmarks, metrics, and workload generators. The workshop focuses on novel cloud properties such as elasticity, performance isolation, dependability, and other non-functional system properties, in addition to classical performance-related metrics such as response time, throughput, scalability, and efficiency. The theme for the 2021 edition is "Benchmarking in the Cloud". HotCloudPerf 2021, co-located with the 12th ACM/SPEC International Conference on Performance Engineering (ICPE 2021), is held on April 19-20th, 2021.

The rapid adoption and the diversification of cloud computing technology exacerbate the importance of a sound experimental methodology for this domain. This work investigates how to measure and report performance in the cloud, and how well the cloud research community is already doing it. We propose a set of eight important methodological principles that combine best-practices from nearby fields with concepts applicable only to clouds, and with new ideas about the time-accuracy trade-off. We show how these principles are applicable using a practical use-case experiment. To this end, we analyze the ability of the newly released SPEC Cloud IaaS benchmark to follow the principles, and showcase real-world experimental studies in common cloud environments that meet the principles. Last, we report on a systematic literature review including top conferences and journals in the field, from 2012 to 2017, analyzing if the practice of reporting cloud performance measurements follows the proposed eight principles. Worryingly, this systematic survey and the subsequent two-round human reviews, reveal that few of the published studies follow the eight experimental principles. We conclude that, although these important principles are simple and basic, the cloud community is yet to adopt them broadly to deliver sound measurement of cloud environments.

No abstract available.

Fast deployment and execution of cloud functions in Function-as-a-Service (FaaS) platforms is critical, for example, for microservices architectures. However, functions that require large packages or libraries are bloated and start slowly. An optimization is to cache packages at the worker nodes instead of bundling them with the functions. However, existing FaaS schedulers are vanilla load balancers, agnostic of packages cached in response to prior function executions, and cannot properly reap the benefits of package caching. We study the case of package-aware scheduling and propose PASch, a novel scheduling algorithm that seeks package affinity during scheduling so that worker nodes can re-use execution environments with preloaded packages. PASch leverages consistent hashing and the power of 2 choices, while actively avoiding worker overload. We implement PASch in a new scheduler for the OpenLambda framework and evaluate it using simulations and real experiments. When using PASch instead of a least loaded balancer, tasks perceive an average speedup of 1.29×, and 80th percentile latency that is 23× faster. Furthermore, for the workload studied in this paper, PASch outperforms consistent hashing with bounded loads-a state-of-the-art load balancing algorithm-yielding a 1.3× average speedup, and a speedup of 1.5× at the 80th percentile.

The proliferation of big data processing platforms has led to radically different system designs, such as MapReduce and the newer Spark. Understanding the workloads of such systems facilitates tuning and could foster new designs. However, whereas MapReduce workloads have been characterized extensively, relatively little public knowledge exists about the characteristics of Spark workloads in representative environments. To address this problem, in this work we collect and analyze a 6-month Spark workload from a major provider of big data processing services, Databricks. Our analysis focuses on a number of key features, such as the long-term trends of reads and modifications, the statistical properties of reads, and the popularity of clusters and of file formats. Overall, we present numerous findings that could form the basis of new systems studies and designs. Our quantitative evidence and its analysis suggest the existence of daily and weekly load imbalances, of heavy-tailed and bursty behaviour, of the relative rarity of modifications, and of proliferation of big data specific formats.

Nowadays, microblog platforms provide a medium to share content and interact with other users. With the large-scale data generated on these platforms, the origin and reasons of users engagement in conversations has attracted the attention of the research community. In this paper, we analyze the factors that might spark conversations in Twitter, for the English and Spanish languages. Using a corpus of 2.7 million tweets, we reconstruct existing conversations, then extract several contextual and content features. Based on the features extracted, we train and evaluate several predictive models to identify tweets that will spark a conversation. Our findings show that conversations are more likely to be initiated by users with high activity level and popularity. For less popular users, the type of content generated is a more important factor. Experimental results shows that the best predictive model is able obtain an average score $F1=0.80$. We made available the dataset scripts and code used in this paper to the research community via Github.

Key-value stores are an important component of cloud applications. These NoSQL databases typically do not support ACID transactions, hence, their applications have diverged from traditional database workloads and are not well represented by benchmarks like TPC-C. In this context, YCSB emerged as the de facto benchmark for cloud serving stores, supporting a wide variety of synthetic workloads. However, no publicly available tool exists that can replay real traces against these systems, this is harmful to the community, as the replay of real workloads is an important system evaluation technique. As a result, others have developed ad-hoc closed-source replay tools that use undocumented replay models, thus making their results impossible to replicate. Furthermore, choosing and implementing the right replay model is not trivial. We propose a trace replay model suitable for key-value stores and describe KV-replay, an open-source replayer that implements this model. We show that using synthetic workloads leads to significant evaluation errors: As much as 33% error in miss rate for small cache sizes and 18% speedup overestimation. Evaluations show that KV-replay is accurate, fast and useful.

Software caches, implemented with in-memory key-value stores, are important components of cloud architectures. In a common scenario, one server may serve requests from several applications with different workloads, each supported by a different backend (database or storage system); these applications compete for an allocation of the total memory. We present a model for dynamic memory partitioning for cloud caches with heterogeneous backends. Our work differs from recent work for cloud caches in that we consider: (1) the effect of having backends with different performance profiles, and (2) the cost of re-partitioning the memory. We discuss implementation issues that must be addressed, including the need for on-line and lightweight mechanisms for estimating the miss rate curves (MRCs) and ways to solve the non-convex optimization problem; specifically, we propose a probabilistic adaptive search algorithm that can be used for discontinuous, non-differentiable, or non-convex MRCs.

Data locality is a fundamental problem to data-parallel applications where data-processing tasks consume different amounts of time and resources at different locations. The problem is especially prominent under stressed conditions such as hot spots. While replication based on data popularity relieves hot spots due to contention for a single file, hot spots caused by skewed node popularity, due to contention for files co-located with each other, are more complex, unpredictable, hence more difficult to deal with. We propose Pandas, a light-weight acceleration engine for data-processing tasks that is robust to changes in load and skewness in node popularity. Pandas is a stochastic delay-optimal algorithm. Trace-driven experiments on Hadoop show that Pandas accelerates the data-processing phase of jobs by 11 times with hot spots and 2.4 times without hot spots over existing schedulers. When the difference in processing times due to location is large, such as applicable to the case of memory-locality, the acceleration by Pandas is 22 times.

This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.

Placing data as close as possible to computation is a common practice of data intensive systems, commonly referred to as the data locality problem. By analyzing existing production systems, we confirm the benefit of data locality and find that data have different popularity and varying correlation of accesses. We propose DARE, a distributed adaptive data replication algorithm that aids the scheduler to achieve better data locality. DARE solves two problems, how many replicas to allocate for each file and where to place them, using probabilistic sampling and a competitive aging algorithm independently at each node. It takes advantage of existing remote data accesses in the system and incurs no extra network usage. Using two mixed workload traces from Face book, we show that DARE improves data locality by more than 7 times with the FIFO scheduler in Hadoop and achieves more than 85% data locality for the FAIR scheduler with delay scheduling. Turnaround time and job slowdown are reduced by 19% and 25\%, respectively.

An alternative to a final programming project in a Distributed Systems course is presented. The alternative project, which can easily be adapted to several Computer Science courses, consists in assigning different course topics to pairs of students, for them to develop an interactive learning object to help their classmates and future students of the class understand the subject. The project was well received by the students of the class, and their comments and survey results suggest that their knowledge on the subject improved both by using the learning objects of their peers and by working in developing their own.

We present the design and implementation of VisFlowConnect, a powerful new tool for visualizing network traffic flow dynamics for situational awareness. The visualization capability provided by VisFlowConnect allows an operator to assess the state of a large and complex network given an overall view of the entire network and filter/drill-down features with a friendly user interface that allows users to request more detailed information of interest such as specific protocol traffic flows. The value of VisFlowConnect specifically for security situational awareness is that any security event, with only a few minor exceptions, is reflected as a traffic flow. Thus in using VisFlowConnect, a user can "see" all security events. We show several experiments in which abnormal behaviors with security implications have been discovered and analyzed using VisFlowConnect. These experiments demonstrate how VisFlowConnect can be a uniquely effective tool to assist security administrators in securing their computer networks.

Intrusion detection is an important part of networked-systems security protection. Although commercial products exist, finding intrusions has proven to be a difficult task with limitations under current techniques. Therefore, improved techniques are needed. We argue the need for correlating data among different logs to improve intrusion detection systems accuracy. We show how different attacks are reflected in different logs and argue that some attacks are not evident when a single log is analyzed. We present experimental results using anomaly detection for the virus Yaha. Through the use of data mining tools (RIPPER) and correlation among logs we improve the effectiveness of an intrusion detection system while reducing false positives.