Modern storage systems orchestrate a group of disks to achieve their performance and reliability goals. Even though such systems are designed to withstand the failure of individual disks, failure of multiple disks poses a unique set of challenges. We empirically investigate disk failure data from a large number of production systems, specifically focusing on the impact of disk failures on RAID storage systems. Our data covers about one million SATA disks from six disk models for periods up to 5 years. We show how observed disk failures weaken the protection provided by RAID. The count ofreallocated sectorscorrelates strongly with impending failures. With these findings we designed RAIDShield, which consists of two components. First, we have built and evaluated an active defense mechanism that monitors the health of each disk and replaces those that are predicted to fail imminently. This proactive protection has been incorporated into our product and is observed to eliminate 88% of triple disk errors, which are 80% of all RAID failures. Second, we have designed and simulated a method of using the joint failure probability to quantify and predict how likely a RAID group is to face multiple simultaneous disk failures, which can identify disks that collectively represent a risk of failure even when no individual disk is flagged in isolation. We find in simulation that RAID-level analysis can effectively identify most vulnerable RAID-6 systems, improving the coverage to 98% of triple errors. We conclude with discussions of operational considerations in deploying RAIDShieldmore broadly and new directions in the analysis of disk errors. One interesting approach is to combine multiple metrics, allowing the values of different indicators to be used for predictions. Using newer field data that reports an additional metric,medium errors, we find that the relative efficacy of reallocated sectors and medium errors varies across disk models, offering an additional way to predict failures.

In order to effectively measure the performance of large scale data management solutions at NetApp, we use a fully automated infrastructure to execute end-to-end system performance tests. Both the software and user requirements of this infrastructure are complex: the system under test runs a multi-protocol, highly specialized operating system and the infrastructure serves a diverse audience of developers, analysts, and field engineers (including both sales and support). In this paper we describe our approach to rapidly constructing automated performance system tests by using a lightweight, little, or domain-specific language called SLSL in order to more effectively express test specifications.

Data centers often consolidate a variety of workloads to increase storage utilization and reduce management costs. Each workload, however, has its own performance targets that need to be met, requiring isolation from the effects of other workloads sharing the system. Satisfying the global throughput and latency targets of each workload is challenging in fully distributed storage systems because workloads can have different data layouts and different requests from the same workload can be serviced by different nodes. Quality of service schemes that manage individual system resources usually rely on resource reservations, often requiring assumptions about the layout of data. On the other hand, solutions for distributed storage tend to treat the storage system as a black box, metering requests issued to the system and often under-utilizing system resources.