With the proliferation of attribute-rich machine-generated data, emerging real-time monitoring, diagnosis, and visualization tools ingest and analyze such data across multiple attributes simultaneously. Due to the sheer volume of the data, applications need storage-efficient and performant data representations to analyze them efficiently.

Object-based cloud storage has been widely adopted for their agility in deploying storage with a very low up-front cost. However, enterprises currently use them to store secondary data and not for expensive primary data. The driving reason is performance; most enterprises conclude that storing primary data in the cloud will not deliver the performance needed to serve typical workloads. Our analysis of real-world traces shows that certain primary data sets can reside in the cloud with its working set cached locally, using a cloud gateway that acts as a caching bridge between local data centers and the cloud. We use a realistic cloud gateway simulator to study the performance and cost of moving such workloads to different cloud backends (like Amazon S3). We find that when equipped with the right techniques, cloud gateways can provide competitive performance and price compared to on-premise storage. We also provide insights on how to build such cloud gateways, especially with respect to caching and prefetching techniques.

Emerging byte-addressable, non-volatile memory like phase-change memory, STT-MRAM, etc. brings persistence at latencies within an order of magnitude of DRAM, thereby motivating their inclusion on the memory bus. According to some recent work on NVM, traditional file systems are ineffective and sub-optimal in accessing data from this low latency media. However, there exists no systematic performance study across different file systems and their various configurations validating this point. In this work, we evaluate the performance of various legacy Linux file systems under various real world workloads on non-volatile memory (NVM) simulated using ramdisk and compare it against NVM optimized file system - PMFS. Our results show that while the default file system configurations are mostly sub-optimal for NVM, these legacy file systems can be tuned using mount and format options to achieve performance that is comparable to NVM-aware file system such as PMFS. Our experiments show that the performance difference between PMFS and ext2/ext3 with execute-in-place (XIP) option is around 5% for many workloads (TPCC and YCSB). Furthermore, based on the learning from our performance study, we present few key file system features such as in-place update layout with XIP, and parallel metadata and data allocations, etc. that could be leveraged by file system designers to improve performance of both legacy and new file systems for NVM.

Enterprise storage systems are facing enormous challenges due to increasing growth and heterogeneity of the data stored. Designing future storage systems requires comprehensive insights that existing trace analysis methods are ill-equipped to supply. In this paper, we seek to provide such insights by using a new methodology that leverages an objective, multi-dimensional statistical technique to extract data access patterns from network storage system traces. We apply our method on two large-scale real-world production network storage system traces to obtain comprehensive access patterns and design insights at user, application, file, and directory levels. We derive simple, easily implementable, threshold-based design optimizations that enable efficient data placement and capacity optimization strategies for servers, consolidation policies for clients, and improved caching performance for both.

Today's Internet services are commonly built over TCP, the standard Internet connection-oriented reliable transport protocol. The endpoint naming scheme of TCP, based on network layer (IP) addresses, creates an implicit binding between a service and the IP address of a server providing it, throughout the lifetime of a client connection. This makes a TCP client prone to all adverse conditions that may affect the server endpoint or the internetwork in between, after the connection is established: congestion or failure in the network, server overloaded, failed or under DoS attack. Studies that quantify the effects of network stability and route availability demonstrate that connectivity failures can significantly impact Internet services. As a result, although highly available servers can be deployed, sustaining continuous service remains a problem. We propose cooperative service model, in which a pool of similar servers, possibly geographically distributed across the Internet, cooperate in sustaining a service by migration of client connections within the pool. The control traffic between servers, needed to support migrated connections, can be carried either over the Internet or over a private network. From client's viewpoint, at any point during the lifetime of its service session, the remote endpoint of its connection may transparently migrate between servers.

We advocate a transport layer protocol for highly available network services by means of transparent migration of the server endpoint of a live connection between cooperating servers that provide the same service. We propose a transport protocol that: (i) offers a better alternative than the simple retransmission to the same server, which may be suffering from overload or a DoS attack, may be down, or may not be easily reachable due to congestion; and (ii) decouples a given service from the unique/fixed identity of its provider. Our protocol can be viewed as an extension to the existing TCP, and compatible with it.