In 2013, AmazonWeb Services revolutionized the data warehousing industry by launching Amazon Redshift, the first fully-managed, petabyte-scale, enterprise-grade cloud data warehouse. Amazon Redshift made it simple and cost-effective to efficiently analyze large volumes of data using existing business intelligence tools. This cloud service was a significant leap from the traditional on-premise data warehousing solutions, which were expensive, not elastic, and required significant expertise to tune and operate. Customers embraced Amazon Redshift and it became the fastest growing service in AWS. Today, tens of thousands of customers use Redshift in AWS's global infrastructure to process exabytes of data daily.

We introduce Ensemble, a runtime framework and associated tools for building application performance models on-the-fly. These dynamic performance models can be used to support complex, highly dimensional resource allocation, and/or what-if performance inquiry in modern heterogeneous environments, such as data centers and Clouds. Ensemble combines simple, partially specified, and lower-dimensionality models to provide good initial approximations for higher dimensionality application performance models. We evaluated Ensemble on industry-standard and scientific applications. The results show that Ensemble provides accurate, fast, and flexible performance models even in the presence of significant environment variability.

As modern multi-tier systems are becoming increasingly large and complex, it becomes more difficult for system analysts to understand the overall behavior of the system, and diagnose performance problems. To assist analysts inspect performance behavior, we introduce SelfTalk, a novel declarative language that allows analysts to query and understand the status of a large scale system. SelfTalk is sufficiently expressive to encode an analyst's high-level hypotheses about system invariants, normal correlations between system metrics, or other a priori derived performance models, such as, "I expect that the throughputs of interconnected system components are linearly correlated". Given a hypothesis, Dena, our runtime support system, instantiates and validates it using actual monitoring data within specific system configurations. We evaluate SelfTalk/Dena by posing several hypotheses about system behavior and querying Dena to validate system behavior in a multi-tier dynamic content server. We find that Dena automatically validates the system performance based on the pre-existing hypotheses and helps to diagnose system misbehavior.

Due to the imperative need to reduce the management costs of large data centers, operators multiplex several concurrent database applications on a server farm connected to shared network attached storage. Determining and enforcing per-application resource quotas in the resulting cache hierarchy, on the fly, poses a complex resource allocation problem spanning the database server and the storage server tiers. This problem is further complicated by the need to provide strict Quality of Service (QoS) guarantees to hosted applications. In this paper, we design and implement a novel coordinated partitioning technique of the database buffer pool and storage cache between applications for any given cache replacement policy and per-application access pattern. We use statistical regression to dynamically determine the mapping between cache quota settings and the resulting per-application QoS. A resource controller embedded within the database engine actuates the partitioning of the two-level cache, converging towards the configuration with maximum application utility, expressed as the service provider revenue in that configuration, based on a set of latency sample points. Our experimental evaluation, using the MySQL database engine, a server farm with consolidated storage, and two e-commerce benchmarks, shows the effectiveness of our technique in enforcing application QoS, as well as maximizing the revenue of the service provider in shared server farms.

The database tier of dynamic content servers at large Internet sites is typically hosted on centralized and expensive hardware. Recently, research prototypes have proposed using database replication on commodity clusters as a more economical scaling solution. In this paper, we propose using database replication to support multiple applications on a shared cluster. Our system dynamically allocates replicas to applications in order to maintain application-level performance in response to either peak loads or failure conditions. This approach allows unifying load and fault management functionality. The main challenge in the design of our system is the lime taken to add database replicas. We present replica allocation policies that take this time delay into account and also design an efficient replica addition method that has minimal impact on other applications.We evaluate our dynamic replication system on a commodity cluster with two standard benchmarks: the TPC-W e-commerce benchmark and the RUBIS auction benchmark. Our evaluation shows that dynamic replication requires fewer resources than static partitioning or full overlap replication policies and provides over 90% latency compliance to each application under a range of load and failure scenarios.

This paper introduces a self-configuring architecture for on-demand resource allocation to applications in a shared database cluster. We use a unified approach to load and fault management based on data replication and reactive replica provisioning. While data replication provides scaling and high availability, reactive provisioning dynamically allocates additional replicas to applications in response to peak loads or failure conditions, thus providing per application performance. We design an efficient method for data migration when joining a new replica to a running application that allows for the quick addition of replicas with minimal disruption of transaction processing. Furthermore, by augmenting the adaptation feedback loop with awareness of the delay introduced by the data migration process in our replicated system, we avoid oscillations in resource allocation.We investigate our transparent database provisioning mechanisms in the context of multitier dynamic content Web servers. We dynamically expand/contract the respective allocations within the database tier for two different applications, the TPC-W e-commerce benchmark and the RUBIS online auction benchmark. We demonstrate that our techniques provide quality of service under different load and failure scenarios.

We consider a cluster architecture in which dynamic content is generated by a database back-end and a collection of Web and application server front-ends. We study the effect of transparent query caching on the performance of such a cluster. Transparency requires that cached entries be invalidated as a result of writes. We start with a coarse-grain table-level automatic invalidation cache. Based on observed workload characteristics, we enhance the cache with the necessary dependency tracking and invalidations at the finer granularity of columns. Finally we reduce the miss penalty of invalidations through full and partial coverage of query results.In terms of system design, a query cache may be located at the database back-end, on dedicated machines, on the front-ends, or on a combination thereof. This paper evaluates the tradeoffs of the different cache designs and the cache location using the TPC-W benchmark.Our experiments show that our transparent query cache improves performance very substantially by up to a factor of 1.5 in throughput and 4.2 in response time overall compared to the baseline table-based invalidation scheme. An important contributor to this end result, our optimization for reducing the miss penalty through full and partial coverage detection of query results from the cache improves response time by up to a factor of 2.9 compared to a cache with fine-grained column-based invalidations alone. Thus, the benefits of the higher hit ratio in our optimizations outweigh the costs of additional processing. The results are less clear-cut in terms of where to locate the cache. Performance differences when varying the cache location and the number of caches are small.