Protecting data from loss is of crucial significance to businesses. Failure to protect data can lead to heavy financial and strategic losses, that are often difficult to recover from. Thus, businesses employ backup techniques to store copies of data to enable failure recovery. But surprisingly, backups often fail. Our analysis of about 48,000 installations over a period of 3 years shows that one in six errors result from inadequate storage capacity and yet, little research has been done in the area of storage capacity forecasting that could mitigate these errors. In this paper, we propose Soothsayer, a simulation model that accurately predicts capacity usage by employing 3 techniques: autoregressive and moving-average modeling, clustering and stochastic modeling, and linear regression. Furthermore, our models provide a range of times when the capacity is likely to be reached, rather than a single point estimate, which is more beneficial in capacity planning. We evaluate the accuracy of our model using synthetic data as well as real world data. Our results show that our models outperform the previous piecewise regression method proposed by Chamness when applied to nonlinear datasets, while performing comparably when applied to linear datasets.

Applications that manage semi-structured data are becoming increasingly commonplace. Current approaches for storing semi-structured data use existing storage machinery; they either map the data to relational databases, or use a combination of flat files and indexes. While employing these existing storage mechanisms provides readily available solutions, there is a need to more closely examine their suitability to this class of data. Particularly, retrofitting existing solutions for semi-structured data can result in a mismatch between the tree structure of the data and the access characteristics of the underlying storage device (disk drive). This study explores various possibilities in the design space of native storage solutions for semi-structured data by exploring alternative approaches that match application data access characteristics to those of the underlying disk drive. For evaluating the effectiveness of the proposed native techniques in relation to the existing solution, we experiment with XML data using the XPathMark benchmark. Extensive evaluation reveals the strengths and weaknesses of the proposed native data layout techniques. While the existing solutions work really well for deep-focused queries into a semi-structured document (those that result in retrieving entire subtrees), the proposed native solutions substantially outperform for the non-deep-focused queries, which we demonstrate are at least as important as the deep-focused. We believe that native data layout techniques offer a unique direction for improving the performance of semi-structured data stores for a variety of important workloads. However, given that the proposed native techniques require circumventing current storage stack abstractions, further investigation is warranted before they can be applied to general-purpose storage systems.

Power consumption within the disk-based storage subsystem forms a substantial portion of the overall energy footprint in commodity systems. Researchers have proposed external caching on a persistent, low-power storage device, which we term external caching device (ECD), to minimize disk activity and conserve energy. While recent simulation-based studies have argued in favor of this approach, the lack of an actual system implementation has precluded answering several key questions about external caching systems. We present the design and implementation of EXCES, an external caching system that employs prefetching, caching, and buffering of disk data for reducing disk activity. EXCES addresses important questions related to external caching, including the estimation of future data popularity, I/O indirection, continuous reconfiguration of the ECD contents, and data consistency. We evaluated EXCES with both micro- and macro- benchmarks that address idle, I/O intensive, and real-world workloads. Overall system energy savings was found to lie in the modest 2–14% range, depending on the workload, in somewhat of a contrast to the higher values predicted by earlier studies. Furthermore, while the CPU and memory overheads of EXCES were well within acceptable limits, we found that flash-based external caching can substantially degrade I/O performance. We believe that external caching systems hold promise. Further improvements in ECD technology, both in terms of their power consumption and performance characteristics can help realize the full potential of such systems.