Benchmark and system parameters often have a significant impact on performance evaluation, which raises a long-lasting question about which settings we should use. This paper studies the feasibility and benefits of extensive evaluation. A full extensive evaluation, which tests all possible settings, is usually too expensive. This work investigates whether it is possible to sample a subset of the settings and, upon them, generate observations that match those from a full extensive evaluation. Towards this goal, we have explored the incremental sampling approach, which starts by measuring a small subset of random settings, builds a prediction model on these samples using the popular ANOVA approach, adds more samples if the model is not accurate enough, and terminates otherwise. To summarize our findings: 1) Enhancing a research prototype to support extensive evaluation mostly involves changing hard-coded configurations, which does not take much effort. 2) Some systems are highly predictable, which means that they can achieve accurate predictions with a low sampling rate, but some systems are less predictable. 3) We have not found a method that can consistently outperform random sampling + ANOVA. Based on these findings, we provide recommendations to improve artifact predictability and strategies for selecting parameter values during evaluation.

With the increasing prevalence of heterogeneous hardware and the increasing need for online reconfiguration, there is increasing demand for heterogeneous configurations. However, allowing different nodes to have different configurations may cause errors when these nodes communicate, even if the configuration of each node uses valid values.

This paper presents the design, implementation, and evaluation of a multi-tiered storage system called MOBBS, which provides hybrid block storage for Virtual Machines (VMs) on top of object-based storage infrastructure. MOBBS is mainly motivated by the gap between the lack of studies on hybrid block storage for VMs and the increasing prevalence of hybrid storage systems. By stripping disk images into partitions and intelligently storing them on different storage tiers according to real-time workload patterns, MOBBS achieves efficient use of multiple storage devices and relieves the burden of data placement. Leveraging the benefits of object-based storage, MOBBS is able to dynamically perform non-disruptive and fine-grained data migration between storage tiers and distribute the complexity of data migration across entire storage nodes. Such designs enable our system to deliver storage for VMs with high scalability and availability under an efficient use of SSDs. We evaluated a Ceph implementation of MOBBS using both block and file system workloads. The results comprehensively demonstrate MOBBS's effectiveness in performance improvement as well as efficient utilization of different storage devices.