S3-compatible object storage has become ubiquitous, used by an ever-expanding range of applications. Workload traces show that many of these applications treat object storage like a traditional file system, with many small reads and writes, yet object storage implementations have not kept up. Optimized for bulk storage, these systems cannot efficiently exploit modern SSDs, requiring large hardware installations to achieve operation rates typical of local file systems on modest machines.

With the increasing dominance of SSDs for local storage, today's network mounted virtual disks can no longer offer competitive performance. We propose a Log-Structured Virtual Disk (LSVD) that couples log-structured approaches at both the cache and storage layer to provide a virtual disk on top of S3-like storage. Both cache and backend store are order-preserving, enabling LSVD to provide strong consistency guarantees in case of failure. Our prototype demonstrates that the approach preserves all the advantages of virtual disks, while offering dramatic performance improvements over not only commonly used virtual disks, but the same disks combined with inconsistent (i.e. unsafe) local caching.

Continuous verification and security analysis of software systems are of paramount importance to many organizations. The state-of-the-art for such operations implements agent-based approaches to inspect the provisioned software stack for security and compliance issues. However, this approach, which runs agents on the systems being analyzed, is vulnerable to some attacks, can incur substantial performance impact, and can introduce significant complexity. In this paper, we present the design and prototype implementation of a general-purpose approach for Non-intrusive Software Introspection (NSI). By adhering to NSI, organizations hosting in the cloud can as well control the software introspection workflow with reduced trust in the provider. Experimental analysis of real-world applications demonstrates that NSI presents a lightweight and scalable approach, and has a negligible impact on the performance of applications running on the instance being introspected.

Shingled Magnetic Recording (SMR) may be combined with conventional (re-writable) recording on the same drive; in host-managed drives shipping today this capability is used to provide a small number of re-writable zones, typically totaling a few tens of GB. Although these re-writable zones are widely used by SMR-aware applications, the literature to date has ignored them and focused on fully-shingled devices. We describe μCache, an SMR translation layer (STL) using re-writable (mutable) zones to take advantage of both workload spatial and temporal locality to reduce the garbage collection overhead resulted from out-of-place writes. In μCache the volume LBA space is divided into fixed -sized buckets and, on write access, the corresponding bucket is copied (promoted) to the re-writable zones, allowing subsequent writes to the same bucket be served in - place resulting in fewer garbage collection cycles. We evaluate μCache in simulation against real-world traces and show that with appropriate parameters it is able to hold the entire write working set of most workloads in re-writable storage, virtually eliminating garbage collection overhead. We also emulate μCache by replaying its translated traces against actual drive and show that 1) it outperforms its examined counterpart, an E-region based translation approach on average by 2x and up to 5.1x, and 2) it incurs additional latency only for a small fraction of write operations, (up to 10%) when compared with conventional non-shingled disks.

Current caching methods for improving the performance of big-data jobs assume high (e.g., full bi-section) bandwidth; however many enterprise data centers and co-location facilities have large network imbalances due to over-subscription and incremental networking upgrades. We describe D3N, a multi-layer cooperative caching architecture that mitigates network imbalances by caching data on the access side of each layer of a hierarchical network topology, adaptively adjusting cache sizes of each layer based on observed workload patterns and network congestion. We have added (and submitted upstream) a 2-layer D3N cache to the Ceph RADOS Gateway; read bandwidth achieves the 5GB/s speed of our SSDs, and we show that it substantially improves big-data job performance while reducing network traffic.

Existing bare-metal cloud services that provide users with physical servers have a number of serious disadvantages over their virtual alternatives, including slow provisioning times, difficulty for users to release servers (physical machines) and then reuse them to handle changes in demand, and poor tolerance to failures. We introduce M2, a bare-metal cloud service that uses network-mounted boot drives to overcome these disadvantages. We describe the architecture and implementation of M2 and compare its agility, scalability and performance to existing systems. We show that M2 can reduce provisioning time by over 50% while offering richer functionality, and comparable run time performance with respect to tools that provision images into local disks. M2 is open source and available at https://github.com/CCI-MOC/ims.

We introduce FSTL, a Framework for Shingled Translation Layers: a toolkit for implementing host-side block translation layers for Shingled Magnetic Recording (SMR) drives. It provides a Linux kernel implementation of key translation mechanisms (write allocation, LBA translation, map persistence, and consistent copying) while allowing translation policy (e.g. layout, cleaning algorithms, crash recovery) to be implemented in a user-space controller which communicates through an ioctl-based API to the kernel data plane. Due to its use of a journaled write format, FSTL-based translation layers are able to handle synchronous and durable writes to random LBAs at high speed. We describe the architecture and implementation of FSTL, and present two FSTL-based translation layers implemented in 400 lines of Python each. Despite the simplicity of the controllers, experiments show our first translation layer performing 1.5x to 10x better than a drive-managed translation layer on trace replay experiments, with performance roughly comparable to the drive-managed device for real file system-based benchmarks; the second translation layer, based on a full-volume extent map, is shown to offer significantly better performance than prior work. Furthermore, we implement and evaluate three cleaning algorithms to demonstrate how FSTL-based translation layers may be readily modified, while still offering the robustness needed for long-duration benchmarks and use.

No abstract available.

Accurately modeling drive-managed Shingled Magnetic Recording (SMR) disks is a challenge, requiring an array of approaches including both existing disk modeling techniques as well as new techniques for inferring internal translation layer algorithms. In this work, we present the first predictive simulation model of a generally available drive-managed SMR disk. Despite the use of unknown proprietary algorithms in this device, our model that is derived from external measurements is able to predict mean latency within a few percent, and with an Root Mean Square (RMS) cumulative latency error of 25% or less for most workloads tested. These variations, although not small, are in most cases less than three times the drive-to-drive variation seen among seemingly identical drives.

We propose a new Exokernel-like layer to allow mutually untrusting physically deployed services to efficiently share the resources of a data center. We believe that such a layer offers not only efficiency gains, but may also enable new economic models, new applications, and new security-sensitive uses. A prototype (currently in active use) demonstrates that the proposed layer is viable, and can support a variety of existing provisioning tools and use cases.

No abstract available.

No abstract available.

We are developing a new public cloud, the Massachusetts Open Cloud (MOC) based on the model of an Open Cloud exchange (OCX). We discuss in this paper the vision of an OCX and how we intend to realize it using the Open Stack open-source cloud platform in the MOC. A limited form of an OCX can be achieved today by layering new services on top of Open Stack. We have performed an analysis of Open Stack to determine the changes needed in order to fully realize the OCX model. We describe these proposed changes, which although significant and requiring broad community involvement will provide functionality of value to both existing single-provider clouds as well as future multi-provider ones.

We introduce Skylight, a novel methodology that combines software and hardware techniques to reverse engineer key properties of drive-managed Shingled Magnetic Recording (SMR) drives. The software part of Skylight measures the latency of controlled I/O operations to infer important properties of drive-managed SMR, including type, structure, and size of the persistent cache; type of cleaning algorithm; type of block mapping; and size of bands. The hardware part of Skylight tracks drive head movements during these tests, using a high-speed camera through an observation window drilled through the cover of the drive. These observations not only confirm inferences from measurements, but resolve ambiguities that arise from the use of latency measurements alone. We show the generality and efficacy of our techniques by running them on top of three emulated and two real SMR drives, discovering valuable performance-relevant details of the behavior of the real SMR drives.

Solid-state drives (SSDs) update data by writing a new copy, rather than overwriting old data, causing prior copies of the same data to be invalidated . These writes are performed in units of pages , while space is reclaimed in units of multipage erase blocks , necessitating copying of any remaining valid pages in the block before reclamation. The efficiency of this cleaning process greatly affects performance under random workloads; in particular, in SSDs, the write bottleneck is typically internal media throughput, and write amplification due to additional internal copying directly reduces application throughput. We present the first nearly-exact closed-form solution for write amplification under greedy cleaning for uniformly-distributed random traffic, validate its accuracy via simulation, and show that its inaccuracies are negligible for reasonable block sizes and overprovisioning ratios. In addition, we also present the first models which predict performance degradation for both LRW (least-recently-written) cleaning and greedy cleaning under simple nonuniform traffic conditions; simulation results show the first model to be exact and the second to be accurate within 2%. We extend the LRW model to arbitrary combinations of random traffic and demonstrate its use in predicting cleaning performance for real-world workloads. Using these analytic models, we examine the strategy of separating “hot” and “cold” data, showing that for our traffic model, such separation eliminates any loss in performance due to nonuniform traffic. We then show how a system which segregates hot and cold data into different block pools may shift free space between these pools in order to achieve improved performance, and how numeric methods may be used with our model to find the optimum operating point, which approaches a write amplification of 1.0 for increasingly skewed traffic. We examine online methods for achieving this optimal operating point and show a control strategy based on our model which achieves high performance for a number of real-world block traces.

In hardware virtualization a hypervisor provides multiple Virtual Machines (VMs) on a single physical system, each executing a separate operating system instance. The hypervisor schedules execution of these VMs much as the scheduler in an operating system does, balancing factors such as fairness and I/O performance. As in an operating system, the scheduler may be vulnerable to malicious behavior on the part of users seeking to deny service to others or maximize their own resource usage. Recently, publically available cloud computing services such as Amazon EC2 have used virtualization to provide customers with virtual machines running on the provider's hardware, typically charging by wall clock time rather than resources consumed. Under this business model, manipulation of the scheduler may allow theft of service at the expense of other customers, rather than merely re-allocating resources within the same administrative domain. We describe a flaw in the Xen scheduler allowing virtual machines to consume almost all CPU time, in preference to other users, and demonstrate kernel-based and user-space versions of the attack. We show results demonstrating the vulnerability in the lab, consuming as much as 98% of CPU time regardless of fair share, as well as on Amazon EC2, where Xen modifications protect other users but still allow theft of service (following the responsible disclosure model, we have reported this vulnerability to Amazon; they have since implemented a fix that we have tested and verified). We provide a novel analysis of the necessary conditions for such attacks, and describe scheduler modifications to eliminate the vulnerability. We present experimental results demonstrating the effectiveness of these defenses while imposing negligible overhead. Also, cloud providers such as Amazon's EC2 do not explicitly reveal the mapping of virtual machines to physical hosts [in: ACM CCS, 2009]. Our attack itself provides a mechanism for detecting the co-placement of VMs, which in conjunction with appropriate algorithms can be utilized to reveal this mapping. Other cloud computing attacks may use this mapping algorithm to detect the placement of victims.

Next generation science will increasingly come to rely on the ability to perform efficient, on-the-fly analytics of data generated by high-performance computing (HPC) simulations, modeling complex physical phenomena. Scientific computing workflows are stymied by the traditional chaining of simulation and data analysis, creating multiple rounds of redundant reads and writes to the storage system, which grows in cost with the ever-increasing gap between compute and storage speeds in HPC clusters. Recent HPC acquisitions have introduced compute node-local flash storage as a means to alleviate this I/O bottleneck. We propose a novel approach, Active Flash, to expedite data analysis pipelines by migrating to the location of the data, the flash device itself. We argue that Active Flash has the potential to enable true out-of-core data analytics by freeing up both the compute core and the associated main memory. By performing analysis locally, dependence on limited bandwidth to a central storage system is reduced, while allowing this analysis to proceed in parallel with the main application. In addition, offloading work from the host to the more power-efficient controller reduces peak system power usage, which is already in the megawatt range and poses a major barrier to HPC system scalability. We propose an architecture for Active Flash, explore energy and performance trade-offs in moving computation from host to storage, demonstrate the ability of appropriate embedded controllers to perform data analysis and reduction tasks at speeds sufficient for this application, and present a simulation study of Active Flash scheduling policies. These results show the viability of the Active Flash model, and its capability to potentially have a transformative impact on scientific data analysis.

Solid state drives (SSDs) update data by writing a new copy, rather than overwriting old data, causing prior copies of the same data to be invalidated. These writes are performed in units of pages, while space is reclaimed in units of multi-page erase blocks, necessitating copying of any remaining valid pages in the block before reclamation. The efficiency of this cleaning process greatly affects performance under random workloads; in particular, in SSDs the write bottleneck is typically internal media throughput, and write amplification due to additional internal copying directly reduces application throughput.

The rising complexity of distributed server applications in Internet data centers has made the tasks of modeling and analyzing their behavior increasingly difficult. This article presents Modellus , a novel system for automated modeling of complex web-based data center applications using methods from queuing theory, data mining, and machine learning. Modellus uses queuing theory and statistical methods to automatically derive models to predict the resource usage of an application and the workload it triggers; these models can be composed to capture multiple dependencies between interacting applications. Model accuracy is maintained by fast, distributed testing, automated relearning of models when they change, and methods to bound prediction errors in composite models. We have implemented a prototype of Modellus, deployed it on a data center testbed, and evaluated its efficacy for modeling and analysis of several distributed multitier web applications. Our results show that this feature-based modeling technique is able to make predictions across several data center tiers, and maintain predictive accuracy (typically 95% or better) in the face of significant shifts in workload composition; we also demonstrate practical applications of the Modellus system to prediction and provisioning of real-world data center applications.

There is a wide gap between the potential performance of NAND flash-based solid state drives (SSDs) and their performance in many real-world applications; understanding this gap requires knowledge of their behavior and internal algorithms for various workloads. We develop analytic models for two commonly-used Flash Translation Layer (FTL) algorithms, as used in SSDs, as well as a methodology for applying these models to real-world workloads. We demonstrate the accuracy of these models via simulation, extend this approach to incorporate measurement-based approximations when detailed parameters are unknown, and validate this methodology against real devices.

Recently, cloud computing services such as Amazon EC2 have used virtualization to provide customers with virtual machines running on the provider's hardware, typically charging by wall clock time rather than resources consumed. Under this business model, manipulation of the scheduler may allow theft-of-service at the expense of other customers. We have discovered and implemented an attack scenario which when implemented on Amazon EC2 allowed virtual machines to consume more CPU time regardless of fair share. We provide a novel analysis of the necessary conditions for such attacks, and describe scheduler modifications to eliminate the vulnerability. We present experimental results demonstrating the effectiveness of these defenses while imposing negligible overhead. Cloud providers such as Amazon's EC2 do not explicitly provide the mapping of VMs to physical hosts. Our attack itself provides a mechanism for detecting the co-placement of VMs, which in conjunction with appropriate algorithms can be utilized to reveal this mapping. We abstract mapping discovery as a problem of finding an unknown partition (i.e. of VMs among physical hosts) using a minimum number of co-location queries. We present an algorithm that is provably optimal when the maximum partition size is bounded. In the unbounded case we show upper and lower bounds using the probabilistic method in conjunction with a sieving technique. Our work has implications beyond this attack, for other cases of system and network topology inference from limited data.

The "Computer Systems" course at Northeastern University is an MS-level core course which attempts to teach students how computers work, through a behavioral approach to the concepts involved in operating systems and their interface to the hardware. As an operating system is typically the first reactive system which students encounter in their studies, the goal of the class is to develop an understanding of the tools and reasoning which are involved in understanding and working with the internals of such a system, whether it be a conventional operating system or (as is more commonly found in industry) a consumer product, networking device, or other embedded system. This course is currently in its third year with enthusiastic responses from students, especially those who have been able to apply its lessons in co-operative work assignments, and an undergraduate class teaching substantially the same material is currently underway.

Many data center virtualization solutions, such as VMware ESX, employ content-based page sharing to consolidate the resources of multiple servers. Page sharing identifies virtual machine memory pages with identical content and consolidates them into a single shared page. This technique, implemented at the host level, applies only between VMs placed on a given physical host. In a multi-server data center, opportunities for sharing may be lost because the VMs holding identical pages are resident on different hosts. In order to obtain the full benefit of content-based page sharing it is necessary to place virtual machines such that VMs with similar memory content are located on the same hosts.

Local storage is required in many sensor network applications, both for archival of detailed event information, as well as to overcome sensor platform memory constraints. Recent gains in energy efficiency of new-generation NAND flash storage have strengthened the case for in-network storage by data-centric sensor network applications. We argue that current storage solutions offering a simple file system abstraction are inadequate for sensor applications to exploit storage. Instead, we propose Capsule—a rich, flexible and portable object storage abstraction that offers stream, file, array, queue and index storage objects for data storage and retrieval. Further, Capsule supports checkpointing and rollback of object state for fault tolerance. Our experiments demonstrate that Capsule provides platform independence, greater functionality and greater energy efficiency than existing storage solutions.

Distributed computation of Voronoi cells in sensor networks, i.e. computing the locus of points in a sensor field closest to a given sensor, is a key building block that supports a number of applications in both the data and control planes. For example, knowledge of Voronoi cells facilitates efficient methods for computing the piece-wise approximation of a field, whereby each sensor acts as a representative for the set of points in its Voronoi cell; awareness of Voronoi boundaries and Voronoi neighbors is also useful in load balancing and energy conservation. The methods currently advocated for distributed Voronoi computation in sensor networks are heuristic approximations that can introduce significant inaccuracies that are difficult to rigorously quantify; we demonstrate that these methods may err by a factor of 5 or more in some circumstances. We present and prove an exact method which eliminates these inaccuracies, at the cost of increased messaging overhead, but without necessitating contact with the entire network. To our knowledge, this is the first distributed algorithm that computes accurate Voronoi cells without requiring all-to-all communication. We implement it as a TinyOS module and quantitatively analyze its performance.

Local storage is required in many sensor network applications, both for archival of detailed event information, as well as to overcome sensor platform memory constraints. While extensive measurement studies have been performed to highlight the trade-off between computation and communication in sensor networks, the role of storage has received little attention. The storage subsystems on currently available sensor platforms have not exploited technology trends, and consequently the energy cost of storage on these platforms is as high as that of communication. Current flash memories, however, offer a low-priced, high-capacity and extremely energy-efficient storage solution.In this paper, we perform a comprehensive evaluation of the active and sleep-mode energy consumption of available flash-based storage options for sensor platforms. Our results demonstrate more than a 100-fold decrease in per-byte energy consumption for surface-mount parallel NAND flash in comparison with the MicaZ on-board serial flash. In addition, this dramatically reduces storage energy costs relative to communication, introducing a new dimension in traditional computation vs communication trade-offs. Our results have significant ramifications on the design of sensor platforms as well as on the energy consumption of sensing applications. We quantify the potential energy gains for two commonly used sensor network services: communication and in-network data aggregation. Our measurements show significant improvements in each service: 50-fold and up to 10-fold reductions in energy for communication and data aggregation respectively.

Improved energy-efficiency and storage capacity of new-generation NAND flash memory makes a compelling case for storage-centric sensor networks. Such a storage-centric sensor network emphasizes the use of platforms with larger storage and more extensive use of the storage capacities on sensors. We demonstrate the feasibility of storage-centric sensor networks using an instance of a storage-centric camera sensor network that is more energy-efficient in comparison to a traditional camera sensor network. We demonstrate multiple camera sensors, each consisting of a Cyclops camera attached to a MicaZ mote, using motion-triggered image capturing. The captured images are archived locally on flash storage and summaries of detected events are transmitted to the base-station. The base-station picks the events of interest from the summaries and requests the original captured image from the sensor as required. The use of high-capacity energy-efficient flash storage at the sensor allows us to trade-off expensive radio communication for cheaper local storage, improving the life-time of the battery and consequently, the life of the storage-centric camera sensor network.

Recent gains in energy-efficiency of new-generation NAND flash storage have strengthened the case for in-network storage by data-centric sensor network applications. This paper argues that a simple file system abstraction is inadequate for realizing the full benefits of high-capacity lowpower NAND flash storage in data-centric applications. Instead we advocate a rich object storage abstraction to support flexible use of the storage system for a variety of application needs and one that is specifically optimized for memory and energy-constrained sensor platforms. We propose Capsule, an energy-optimized log-structured object storage system for flash memories that enables sensor applications to exploit storage resources in a multitude of ways. Capsule employs a hardware abstraction layer that hides the vagaries of flash memories for the application and supports energy-optimized implementations of commonly used storage objects such as streams, files, arrays, queues and lists. Further, Capsule supports checkpointing and rollback of object states to tolerate software faults in sensor applications running on inexpensive, unreliable hardware. Our experiments demonstrate that Capsule provides platform-independence, greater functionality, more tunability, and greater energy-efficiency than existing sensor storage solutions, while operating even within the memory constraints of the Mica2 Mote. Our experiments not only demonstrate the energy and memory-efficiency of I/O operations in Capsule but also shows that Capsule consumes less than 15% of the total energy cost in a typical sensor application.

Archival storage of sensor data is necessary for applications that query, mine, and analyze such data for interesting features and trends. We argue that existing storage systems are designed primarily for flat hierarchies of homogeneous sensor nodes and do not fully exploit the multi-tier nature of emerging sensor networks, where an application can comprise tens of tethered proxies, each managing tens to hundreds of untethered sensors. We present TSAR, a fundamentally different storage architecture that envisions separation of data from metadata by employing local archiving at the sensors and distributed indexing at the proxies. At the proxy tier, TSAR employs a novel multi-resolution ordered distributed index structure, the Interval Skip Graph, for efficiently supporting spatio-temporal and value queries. At the sensor tier,TSAR supports energy-aware adaptive summarization that can trade off the cost of transmitting metadata to the proxies against the overhead of false hits resulting from querying a coarse-grain index. We implement TSAR in a two-tier sensor testbed comprising Stargate-based proxies and Mote-based sensors. Our experiments demonstrate the benefits and feasibility of using our energy-efficient storage architecture in multi-tier sensor networks.