Hybrid cloud is an integrated cloud computing environment utilizing a mix of public cloud, private cloud, and on-premise IT infrastructures. Workload awareness, defined as a detailed full range understanding of each individual workload, is essential in implementing the hybrid cloud. While it is critical to perform an accurate analysis to determine which workloads are appropriate for on-premise deployment versus which workloads can be migrated to a cloud off-premise, the assessment is mainly performed by rule or policy based approaches. In this paper, we introduce StackInsights, a novel cognitive system to automatically analyze and predict the cloud readiness of workloads for an enterprise. Our system harnesses the critical metrics across the entire stack: (1) infrastructure metrics, (2) data relevance metrics, and (3) application taxonomy, to identify workloads that have characteristics of (a) low sensitivity with respect to business security, criticality and compliance, and (b) low response time requirements and access patterns. Since the capture of the data relevance metrics involves an intrusive and in-depth scanning of the content of storage objects, a machine learning model is applied to perform the business relevance classification by learning from the meta level metrics harnessed across stack. In contrast to traditional methods, StackInsights significantly reduces the total time for hybrid cloud readiness assessment by orders of magnitude.

With the wide adoption of hybrid cloud, there are many potential risks that need to be mitigated to ensure that the utilizations of services are at their optimal levels. One of the major risks that has garnered much attention is maintaining maximum security and confidentiality for sensitive information. Detecting sensitive content at near real-time and at cloud scale has become a critical first step for organizations to prevent data loss and comply with data privacy laws and regulations. Proactive detection raises security awareness at the early stage and thus can be used to govern how the information should be managed, protected, and utilized in the hybrid cloud. In contrast to traditional dictionary or policy-based approaches, we introduce a system that detects sensitive content by leveraging its semantic contextual information through various machine learning and deep learning techniques at different levels of granularity within the document, and is the first of its kind.

Data protection is the process of backing up data in case of a data loss event. It is one of the most critical routine activities for every organization. Detecting abnormal backup jobs is important to prevent data protection failures and ensure the service quality. Given the large scale backup endpoints and the variety of backup jobs, from a backup-as-a-service provider viewpoint, we need a scalable and flexible outlier detection method that can model a huge number of objects and well capture their diverse patterns. In this paper, we introduce H2O, a novel hybrid and hierarchical method to detect outliers from millions of backup jobs for large scale data protection. Our method automatically selects an ensemble of outlier detection models for each multivariate time series composed by the backup metrics collected for each backup endpoint by learning their exhibited characteristics. Interactions among multiple variables are considered to better detect true outliers and reduce false positives. In particular, a new seasonal-trend decomposition based outlier detection method is developed, considering the interactions among variables in the form of common trends, which is robust to the presence of outliers in the training data. The model selection process is hierarchical, following a global to local fashion. The final outlier is determined through an ensemble learning by multiple models. Built on top of Apache Spark, H2O has been deployed to detect outliers in a large and complex data protection environment with more than 600,000 backup endpoints and 3 million daily backup jobs. To the best of our knowledge, this is the first work that selects and constructs large scale outlier detection models for multivariate time series on Big Data platforms.

Multi-tenant storage management environments typically manage multiple enterprise accounts with heterogeneous storage footprints. Identifying and grouping accounts with similar storage footprints into clusters reduces account management overhead, and provides a framework for data-driven storage recommendation services. This paper describes a method for the clustering of accounts in multi-tenant storage management environments. Storage system vendors, models, and footprints are captured as a set of properties, and a pairwise distance function is employed to grade similarity between accounts along multiple dimensions. A graph representation is created based on account similarity values. Finally, the clustering algorithm is defined as a graph algorithm with the purpose of repeatedly finding maximum cliques, and removing them from the graph. The result of the graph algorithm is a set of clusters, each grouping together accounts with very similar storage footprints. Clusters are then rated along multiple metrics, are compared to their peers along multiple performance dimensions, and receive recommendations on how to further improve storage efficiency.

Frequent Item set Mining (FIM) is a classic data mining topic with many real world applications such as market basket analysis. Many algorithms including Apriori, FP-Growth, and Eclat were proposed in the FIM field. As the dataset size grows, researchers have proposed MapReduce version of FIM algorithms to meet the big data challenge. This paper proposes new improvements to the MapReduce implementation of FIM algorithm by introducing a cache layer and a selective online analyzer. We have evaluated the effectiveness and efficiency of Smart Cache via extensive experiments on four public datasets. Smart Cache can reduce on average 45.4%, and up to 97.0% of the total execution time compared with the state-of-the-art solution.

Summary form only given. Emergence and adoption of cloud computing have become widely prevalent given the value proposition it brings to an enterprise in terms of agility and cost effectiveness. Big data analytical capabilities (specifically treating storage/system management as a big data problem for a service provider) using Cloud delivery models is defined as Analytics as a Service or Software as a Service. This service simplifies obtaining useful insights from an operational enterprise data center leading to cost and performance optimizations.Software defined environments decouple the control planes from the data planes that were often vertically integrated in a traditional networking or storage systems. The decoupling between the control planes and the data planes enables opportunities for improved security, resiliency and IT optimization in general. This talk describes our novel approach in hosting the systems management platform (a.k.a. control plane) in the cloud offered to enterprises in Software as a Service (SaaS) model. Specifically, in this presentation, focus is on the analytics layer with SaaS paradigm enabling data centers to visualize, optimize and forecast infrastructure via a simple capture, analyze and govern framework. At the core, it uses big data analytics to extract actionable insights from system management metrics data. Our system is developed in research and deployed across customers, where core focus is on agility, elasticity and scalability of the analytics framework. We demonstrate few system/storage management analytics case studies to demonstrate cost and performance optimization for both cloud consumer as well as service provider. Actionable insights generated from the analytics platform are implemented in an automated fashion via an OpenStack based platform.

Storage management aims to improve the data center performance by optimizing the underlying storage resources more efficiently. The advent of cloud computing technologies introduces a paradigm shift from conventional on-premise storage management solutions to multitenant storage management as a service models. The advantage of centralized multitenant storage management platforms lies in the integrated procedures and the unified platform of data collection, governance, and analytics to gauge the effectiveness and efficiency of tenant storage environments. In this paper, we introduce our data-driven storage optimization framework, which is a centralized storage data analytics module to provide recommendations for the storage administrator. We use three example use cases to illustrate the exploitation of individual data center operational data to obtain actionable insights for better storage management solutions.

Continuous Data Protection (CDP) enables recoverability to any point in time (time travel) facilitated via journaling of every write made by a system to disk. Stringent storage performance and capacity requirements for journaling make CDP a very high cost solution leading to limited adoption. In this work we first explore the feasibility of building such a CDP function on top of cheap commodity storage exposed via cloud object stores. Based on this analysis, we propose cCDP - a Cloud Continuous Data Protection framework that efficiently combines cloud object stores with edge caching to address requirements of low cost, high capacity, low latency and high storage throughput. cCDP with careful tuning can not only meet but surpasses the write throughput and latency requirements of CDP with minimal buffer overheads (<; 1%). Recovery lookup performance (Identification & Ordering) of cCDP with hybrid data layout is within 28% of Btree based temporal data layout and is about 52.7% better than the naive data layout with name encoding with significantly lower edge buffer and write throughput overheads than the Btree based temporal data layout.

In enterprise data centers, reliable performance models on storage devices are desirable for efficient storage management and optimization. However, many cloud environments consist of heterogeneous storage devices, e.g., a mixture of commodity disks, where accurate performance models are of particular challenge to attain. In this paper, we propose a lightweight queueing-based storage performance modeling framework, which is able to infer the maximum IO load that a storage device can sustain, as well as its IO load v.s. response time performance curve. Our inference framework views the underlying storage resources as blackboxes and only utilizes historical measurements of the IO and response time on the devices. In an OpenStack environment, we also develop a new storage volume placement algorithm using our performance inference and modeling framework. Experimental results show that our solution can provide up to 80% increase of the IO throughput, in tandem with a 40% reduction of the average response time, compared to the performance provided by the default OpenStack policy.

Virtualization is a double-edged sword in large scale cloud computing environments. On one hand, virtualization provides a logical and unified view of the underlying cloud resources to facilitate more efficient resource utilization and to support multi-tenancy. On the other hand, virtualization introduces additional layers of indirections which make the virtual-to-physical resource mapping relationship obscure, or prohibitively costly to unveil, especially with multiple layers of virtualization at presence, e.g., server virtualization, network virtualization, and storage virtualization. In this work, we propose a virtual-to-physical mapping inference framework by analyzing the I/O activities of inputs and outputs of the virtualization layer. Using a virtualized storage example, we show that our lightweight solution can unveil the virtual-to-physical resource mapping relationship in a non-intrusive fashion, which can be used in many cloud management tasks such as root cause analysis and diagnosis.

Deduplication is a commonly-used technique on disk-based storage pools. However, deduplication has not been used for tape-based pools: tape characteristics, such as high mount and seek times combined with data fragmentation resulting from deduplication create a toxic combination that leads to unacceptably high retrieval times. This work proposes DedupT, a system that efficiently supports deduplication on tape pools. This paper (i) details the main challenges to enable efficient deduplication on tape libraries, (ii) presents a class of solutions based on graph-modeling of similarity between data items that enables efficient placement on tapes; and (iii) presents the design and evaluation of novel cross-tape and on-tape chunk placement algorithms that alleviate tape mount time overhead and reduce on-tape data fragmentation. Using 4.5 TB of real-world workloads, we show that DedupT retains at least 95% of the deduplication efficiency. We show that DedupT mitigates major retrieval time overheads, and, due to reading less data, is able to offer better restore performance compared to the case of restoring non-deduplicated data.

Data protection using backup is one of the most critical IT management operations to ensure business continuity, which is also constantly evolving due to the emerging challenges in the “Big Data Era.” In this paper, we introduce our ongoing research effort in designing intelligent enterprise backup management solutions by obtaining actionable insights from voluminous backup job metadata across data centers. Our proof-of-concept scalable data analytics framework integrated with enterprise backup architecture is introduced, and use case studies of leveraging big data analytics on enterprise backup framework are discussed.

Intelligent workload consolidation in storage systems leads to better Return On Investment (ROI), in terms of more efficient use of data center resources, better Quality of Service (QoS), and lower power consumption. This is particularly significant yet challenging in a cloud environment, in which a large set of different workloads multiplex on a shared, heterogeneous infrastructure. However, the increasing availability of fine grained workload logging facilities allows better insights to be gained from workload profiles. As a consequence, consolidation can be done more deeply, according to a detailed understanding of how well given workloads mix. We describe IO Tetris, which takes a first look at fine-grained consolidation in large-scale storage systems by leveraging temporal patterns found in real-world I/O traces gathered from enterprise storage environments. The core functionality of IO Tetris consists of two stages. A grouping stage performs hierarchical grouping of storage workloads to find complementary groupings that consolidate well together over time and conflicting ones that do not. After that, a migration stage examines the discovered groupings to determine how to maximize resource utilization efficiency while minimizing migration costs. Experiments based on customer I/O traces from a high-end enterprise class IBM storage controller show that a non-trivial number of IO Tetris groupings exist in real-world storage workloads, and that these groupings can be leveraged to achieve better storage consolidation in a cloud setting.

SUMMARY The proliferation of cloud services and other forms of service‐oriented computing continues to accelerate. Alongside this development is an ever‐increasing need for storage within the data centres that host these services. Management applications used by cloud providers to configure their infrastructure should ideally operate in terms of high‐level policy goals, and not burden administrators with the details presented by particular instances of storage systems. One common technology used by cloud providers is the Storage Area Network (SAN). Support for seamless scalability is engineered into SAN devices. However, SAN infrastructure has a very large parameter space: their optimal deployment is a difficult challenge, and subsequent management in cloud storage continues to be difficult. parindent = 10pt In this article, we discuss our work in SAN configuration middleware, which aims to provide users of large‐scale storage infrastructure such as cloud providers with tools to assist them in their management and evolution of heterogeneous SAN environments. We propose a middleware rather than a stand‐alone tool so that the middleware can be a proxy for interacting with, and informing, a central repository of SAN configurations. Storage system users can have their SAN configurations validated against a knowledge base of best practices that are contained within the central repository. Desensitized information is exported from local management applications to the repository, and the local middleware can subscribe to updates that proactively notify storage users should particular configurations be updated to be considered as sub‐optimal, or unsafe. Copyright © 2011 John Wiley & Sons, Ltd.

Automatic discovery and monitoring of IT resources is a critical part of enterprise systems management. In addition to ascertaining internal device configurations, this discovery process may also need to capture the capabilities, usage, connectivity, availability, and other information related to various IT components. Systems resource management (SRM) tools typically implement this discovery process using device specific APIs, custom agents and/or some standard-based solution (like WBEM and CIM). The discovery actions need to be systematically planned; an inefficient implementation or scheduling may easily take from a few minutes to several hours to complete in a large heterogeneous enterprise datacenter. This paper discusses the various challenges associated in discovering the configuration of a datacenter environment and presents an autonomic configuration monitoring middleware called Magellan that builds upon industry best practices and standards. Magellan reduces the overall discovery time by more than 50% in our micro-benchmark experiments as well as in a large datacenter configuration of a major financial organization.

Clustering solutions are frequently used in large enterprise and mission critical applications with high performance and availability requirements. This is achieved by deploying multiple servers for the purpose of load balancing or by having redundant servers for the purpose of high availability. Most cluster management tools focus on the availability of servers and IP connectivity, and rely on shared network storage like Network Attached Storage (NAS), Storage Area Network (SAN), etc. for storage resiliency. Modern enterprise storage systems are complex, with millions of possible configurations. For end-to-end performance and availability, the configuration of storage resources and interconnection fabric need to be carefully planned. Adhoc configuration may lead to poor availability, deteriorated performance or higher cost. In this paper, we discuss the challenges associated with provisioning storage resources in cluster environment and present a list of best practices and novel optimization strategies to make storage more resilient for clusters. We have build a tool called Casper that automates the process of planning, optimization and deployment of shared network storage for clusters. Casper distributes storage load across different storage nodes and across different network paths to ensure load balancing and prevent single point of failure.

We present the design and implementation of ENDEAVOUR - a framework for integrated end-to-end disaster recovery (DR) planning. Unlike existing research that provides DR planning within a single layer of the IT stack (e.g. storage controller based replication), ENDEAVOUR can choose technologies and composition of technologies across multiple layers like virtual machines, databases and storage controllers. ENDEAVOUR uses a canonical model of available replication technologies at all layers, explores strategies to compose them, and performs a novel map-search-reduce heuristic to identify the best DR plans for given administrator requirements. We present a detailed analysis of ENDEAVOUR including empirical characterization of various DR technologies, their composition, and a end-to-end case study.

Operational cost can account up to 70% of the total IT budget of an enterprise. Total Cost of Ownership (TCO) for an enterprise scale data center is significantly dependent upon type and quality of the system management solutions deployed. Providing an uniform management solution for heterogeneous set of devices interconnected inside and across data centers that are geographically distributed is a major requirement from all large scale enterprises. Given the limited resources of testing infrastructure, verification of proper operation as well as automated validation of system management software poses a significant challenge. In this paper, we propose a simulation framework, namely VLS (Virtual Lab. Service), offered as a service to address this challenge. Our tool provides simulation for a wide variety of data center configurations that are used by system management solution developers and testers. VLS provides simulated IT infrastructure based on declarative input specification while keeping in mind the current paradigm of sharing and reservation mechanism followed by teams to access lab. equipments over Wide Area Network (WAN).We deployed this service in our lab. environment for testing of two popular storage management suites and present one of our case study to demonstrate these capabilities.

The complexity of modern data centers has evolved significantly in recent years. One typically is comprised of a large number and types of middleware and applications that are hosted in a heterogeneous pool of both physical and virtual servers, connected by a complex web of virtual and physical networks. Therefore, to manage everything in a data center, system administrators usually need a plethora of management tools since one tool often manages only one type of devices. The boundaries between the different management tools can limit productivity of system administrators on their daily tasks as each tool only offers a partial view of the entire managed environment. As a result, advanced analytics such as impact analysis and problem determination are generally not achievable using the traditional management tools as they require a holistic view of the entire data center. In this paper, we describe an integrated management system for applications, servers, network and storage devices called DataGraph. Our system integrates data across heterogeneous point products and agents for management and monitoring to enable the above mentioned management analytics capabilities. A common data model is introduced to federate data collected by the different tools in multiple database repositories so no modifications are needed to existing management tools. A common integrated web user interface is implemented to facilitate management tasks that would otherwise require invoking multiple tools. We deployed this tool in a lab environment and demonstrated these analytics capabilities through several case studies.

Many enterprise environments have databases running on network-attached storage infrastructure (referred to as Storage Area Networks or SANs ). Both the database and the SAN are complex subsystems that are managed by separate teams of administrators. As often as not, database administrators have limited understanding of SAN configuration and behavior, and limited visibility into the SAN's run-time performance; and vice versa for the SAN administrators. Diagnosing the cause of performance problems is a challenging exercise in these environments. We propose to remedy the situation through a novel tool, called Diads, for database and SAN problem diagnosis. This demonstration proposal summarizes the technical innovations in Diads: (i) a powerful abstraction called Annotated Plan Graphs (APGs) that ties together the execution path of queries in the database and the SAN using low-overhead monitoring data, and (ii) a diagnosis workflow that combines domain-specific knowledge with machine-learning techniques. The scenarios presented in the demonstration are also described.

Most organizations are becoming increasingly reliant on IT product and services to manage their daily operations. The total cost of ownership (TCO), which includes the hardware and software purchase cost, management cost, etc., has significantly increased and forms one of the major portions of the total expenditure of the company. CIOs have been struggling to justify the increased costs and at the same time fulfill the IT needs of their organizations. For businesses to be successful, these costs need to be carefully accounted and attributed to specific processes or user groups/departments responsible for the consumption of IT resources. This process is called IT chargeback and although desirable, is hard to implement because of the increased consolidation of IT resources via technologies like virtualization. Current IT chargeback methods are either too complex or too adhoc, and often a times lead to unnecessary tensions between IT and business departments and fail to achieve the goal for which chargeback was implemented. This paper presents a new tool called ChargeView that automates the process of IT costing and chargeback. First, it provides a flexible hierarchical framework that encapsulates the cost of IT operations at different level of granularity. Second, it provides an easy way to account for different kind of hardware and management costs. Third, it permits implementation of multiple chargeback policies that fit the organization goals and establishes relationship between the cost and the usage by different users and departments within an organization. Finally, its advanced analytics functions can keep track of usage and cost trends, measure unused resources and aid in determining service pricing. We discuss the prototype implementation of ChargeView and show how it has been used for managing complex systems and storage networks.