Distributed tracing has become an essential tool to navigate performance of complex, distributed cloud-native applications, providing a comprehensive view of a request from end-to-end. However, the sheer amount of data generated by distributed tracing can be overwhelming, making it difficult to store, process, and extract meaningful insights. This paper presents the vision of ’nuffTrace that embodies a novel tree-based probabilistic data structure that summarizes trace data in a compact form without storing all of the data, enabling developers to analyze cloud application performance with high accuracy and efficiency.

Online experimentation is an agile software development practice that plays an essential role in enabling rapid innovation. Existing solutions for online experimentation in Web and mobile applications are unsuitable for cloud applications. There is a need for rethinking online experimentation in the cloud to advance the state-of-the-art by considering the unique challenges posed by cloud environments.

Virtualization promises to reduce data centers' total cost of ownership by enabling the creation of a small set of standardized building blocks to be shared and used many times indifferent software stacks. However, without proper methodology and tools, an organization can easily end up with a large number of one-off virtual images, adversely affecting the cost. We propose an approach, tool, and algorithms for constructing high-quality, semantically-rich image building blocks that are easy to share, compose, and reuse. In our approach, domain experts codify knowledge of a particular software product (or a combination thereof) in a platform- and cloud-agnostic software bundle. Image builders easily construct virtual images by composing a set of standardized bundles. Semantic-based validation guarantees a valid and complete image design. Moreover, we propose algorithms to automate image design by searching for an optimal set of building blocks taking into account multiple metrics such as cost, size, and expected build duration.

In this paper, we propose a management framework for protecting large computer systems against operator mistakes. By detecting and confining mistakes to isolated portions of the managed system, our framework facilitates correct operation even by inexperienced operators. We built a prototype management system called Barricade based on our framework. We evaluate Barricade by deploying it for two different systems, a prototype Internet service and an enterprise computer infrastructure, and conducting experiments with 20 volunteer operators. Our results are very promising. For example, we show that Barricade can detect and contain 39 out of the 43 mistakes that we observed in 49 live operator experiments performed with our Internet service.

Operator mistakes are a significant source of unavailability in Internet services. In our previous work, we proposed operator action validation as an approach for detecting mistakes while hiding them from the service and its users. Previous validation strategies have limitations, however, including the need for instances of correct behavior for comparison. In this paper, we propose a novel model-based validation strategy that addresses these limitations and complements our previous techniques. Model-based validation calls for service engineers to define models of Internet services that can be used to differentiate between correct and incorrect configurations and behaviors. These models are then used to guide the specification of validation assertions that check the correctness of operator actions before they are exposed. We have implemented a prototype model-based validation system for two services, the Web crawler of a commercial search engine (Ask.com) and an academic yet realistic online auction service. Experimentation with model-based validation demonstrates that it is highly effective at detecting and hiding both activated and latent mistakes.

The complexity of today’s computer systems poses a challenge to system administrators. Current systems comprise a multitude of inter-related software components running on different servers. In this paper, we propose the use of the stackable storage mechanism as the foundation of centralized systems management. At the management level, we show how this mechanism can be used to implement an infrastructure that allows administrators to perform typical tasks fast and effortlessly. In particular, we find that our prototype could have avoided 40% of the human mistakes observed experimentally by previous research. At the storage level, we identify three key characteristics of stackable storage that allow the definition of different policies with distinct performance and scalability behaviors. We quantitatively compare five storage policies under different workloads and conclude that stackable storage is a viable approach.

Operator mistakes have been identified as a significant source of unavailability in Internet services. In this paper, we propose a new language, A, for service engineers to write assertions about expected behaviors, proper configurations, and proper structural characteristics. This formalized specification of correct behavior can be used to bolster system understanding, as well as help to flag operator mistakes in a distributed system. Operator mistakes can be caused by anything from static misconfiguration to physical placement of wires and machines. This language, along with its associated runtime system, seeks to be flexible and robust enough to deal with the wide array of operator mistakes while maintaining a simple interface for designers or programmers.

Online services are rapidly becoming the supporting infrastructure for numerous users' work and leisure, placing higher demands on their availability and correct functioning. Increasingly, these services are comprised of complex conglomerates of distributed hardware and software components. Added to this complexity, these services evolve quite frequently accumulating considerable heterogeneity within them, while allowing little time for their in-depth understanding by service personnel. Thus, it is not surprising that mistakes by service operators are common, and have been deemed to be the primary cause of service downtime.