Cloud Causality Analyzer (CCA) is an ML-based analytical pipeline to automate the tedious process of Root Cause Analysis (RCA) of Cloud IT events. The 3-stage pipeline is composed of 9 functional modules, including dimensionality reduction (feature engineering, selection and compression), embedded anomaly detection, and an ensemble of 3 custom explainability and causality models for Cloud Key Performance Indicators (KPI). Our challenge is: How to apply a reduced (sub)set of judiciously selected KPIs to detect Cloud performance anomalies, and their respective root causal culprits, all without compromising accuracy?

Anomaly detection in data storage systems is a challenging problem due to the high dimensional sequential data involved, and lack of labels. The state of the art for automating anomaly detection in these systems typically relies on hand crafted rules and thresholds which mainly allow to distinguish between normal and abnormal behavior of each indicator in isolation. In this work we present an end-to-end framework based on convolutional autoencoders which not only allows for anomaly detection on multivariate time series data, but also provides explainability. This is done by identifying similar historic anomalies and extracting the most influential indicators. These are then presented to relevant personnel such as system designers and architects, or to support engineers for further analysis. We demonstrate the application of this framework along with an intuitive interactive web interface which was developed for data storage system anomaly detection. We discuss how this framework along with its explainability aspects enables support engineers to effectively tackle abnormal behaviors, all while allowing for crucial feedback.

Detecting anomalies from high-dimensional multivariate temporal data is challenging, because of the non-linear, complex relationships between signals. Recently, deep learning methods based on autoencoders have been shown to capture these relationships and accurately discern between normal and abnormal patterns of behavior, even in fully unsupervised scenarios. However, validating the anomalies detected is difficult without additional explanations. In this paper, we extend SHAP -- a unified framework for providing additive explanations, previously applied for supervised models -- with influence weighting, in order to explain anomalies detected from multivariate time series with a GRU-based autoencoder. Namely, we extract the signals that contribute most to an anomaly and those that counteract it. We evaluate our approach on two use cases and show that we can generate insightful explanations for both single and multiple anomalies.

In this work we present MTEX-CNN, a novel explainable convolutional neural network architecture which can not only be used for making predictions based on multivariate time series data, but also for explaining these predictions. The network architecture consists of two stages and utilizes particular kernel sizes. This allows us to apply gradient based methods for generating saliency maps for both the time dimension and the features. The first stage of the architecture explains which features are most significant to the predictions, while the second stage explains which time segments are the most significant. We validate our approach on two use cases, namely to predict rare server outages in the wild, as well as the average energy production of photovoltaic power plants based on a benchmark data set. We show that our explanations shed light over what the model has learned. We validate this by retraining the network using the most significant features extracted from the explanations and retaining similar performance to training with the full set of features.

Failures of memory modules have been a concern for a long time, as they are costly both in terms of hardware replacement and service disruption. These failures can be preceded by correctable (soft) and then uncorrectable (hard) errors, which accumulate over time. Valuable large scale studies of DIMM errors in the wild [2, 1] analyze in depth hard and soft errors and their correlations with specific sensors. However, little has been reported on how these findings could be used to automatically predict future DIMM failures. We show that by understanding which factors drive such failures, we can build intelligent predictive models with off-the-shelf machine learning techniques to predict DIMM failures ahead of time with high accuracy. Such models not only provide early signs of failures, but also allow administrators to proactively replace DIMMs at risk weeks in advance, thus avoiding "memory loss" of their commodity hardware.

Disks are among the most frequently failing components in today's IT environments. Despite a set of defense mechanisms such as RAID, the availability and reliability of the system are still often impacted severely. In this paper, we present a highly accurate SMART-based analysis pipeline that can correctly predict the necessity of a disk replacement even 10-15 days in advance. Our method has been built and evaluated on more than 30000 disks from two major manufacturers, monitored over 17 months. Our approach employs statistical techniques to automatically detect which SMART parameters correlate with disk replacement and uses them to predict the replacement of a disk with even 98% accuracy.

Configuring enterprise database management systems is a notoriously hard problem. The combinatorial parameter space makes it intractable to run and observe the DBMS behavior in all scenarios. Thus, the database administrator has the difficult task of choosing DBMS configurations that potentially lead to critical incidents, thus hindering its availability or performance. We propose using machine learning to understand how configuring a DBMS can lead to such high risk incidents. We collect historical data from three IT environments that run both IBM DB2 and Oracle DBMS. Then, we implement several linear and non-linear multivariate models to identify and learn from high risk configurations. We analyze their performance, in terms of accuracy, cost, generalization and interpretability. Results show that high risk configurations can be identified with extremely high accuracy and that the database administrator can potentially benefit from the rules extracted to reconfigure in order to prevent incidents.

With the advancement of relational databases, the number of configuration parameters that control memory allocation, concurrency, cost of query plans, I/O optimization, logging, recovery or transaction consistency, increases. Users and even expert database administrators struggle to tune these parameters in order to ensure high availability and performance, and in many cases rely on their experience and some rules of thumb. Research on improving database manageability has shown that this is a critical, but hard problem. In this paper, we propose a highly accurate multivariate statistical model that identifies databases which are bound to raise high volumes of incidents over time. Moreover, we show that by adding detailed configuration parameters to the model, we can better link the problems reported in incident tickets to specific poor database configurations. Finally, we analyze trends of top-ranked parameters and compare their values between problematic and non-problematic databases, in order to suggest better configurations.

We present a novel technique that optimizes the dispatching of incident tickets to the agents in an IT Service Support Environment. Unlike the common skill-based dispatching, our approach also takes empirical evidence on the agent's speed from historical data into account. Our solution consists of two parts. First, a novel technique clusters historic tickets into incident categories that are discriminative in terms of agent's performance. Second, a dispatching policy selects, for an incoming ticket, the fastest available agent according to the target cluster. We show that, for ticket data collected from several Service Delivery Units, our new dispatching technique can reduce service time between $35\%$ and $44\%$.

A company's IT infrastructure delivers the basic hardware, networking, operating system, and middleware support to the business' applications. IT service providers perform incident and problem resolution, as well as user administration and change implementation required to maintain the availability and service provided for the business. As a result, they become increasingly challenged with delivering better, faster, and cheaper services to their customers. With the variety of incident tickets reported on a daily basis, understanding where and how much effort is spent to resolve them is critical. Moreover, analyzing the effort data identifies opportunities for self-service and automation, as well as what modernization strategies businesses should implement to reduce incident volumes and, by association, labor effort. In this paper, we conduct a large scale study on the incident and server factors that affect technician effort and quantify their impact. We show that the nature of the incidents and their complexity, the assigned support groups, as well as the underlying OS type play a major role in how much labor effort is spent towards resolving such tickets.

In today's commercial data centers, the computation density grows continuously as the number of hardware components and workloads in units of virtual machines increase. The service availability guaranteed by data centers heavily depends on the reliability of the physical and virtual servers. In this study, we conduct an analysis on 10K virtual and physical machines hosted on five commercial data centers over an observation period of one year. Our objective is to establish a sound understanding of the differences and similarities between failures of physical and virtual machines. We first capture their failure patterns, i.e., the failure rates, the distributions of times between failures and of repair times, as well as, the time and space dependency of failures. Moreover, we correlate failures with the resource capacity and run-time usage to identify the characteristics of failing servers. Finally, we discuss how virtual machine management actions, i.e., consolidation and on/off frequency, impact virtual machine failures.

Technology refresh is an important component in data center management. The goal of this paper is to assess the impact of HW and OS currency on server availability based on a large set of incident tickets and server attributes data collected from several different IT environments. In order to achieve this we first identify the server failure incidents using a machine learning method for automatic ticket classification. Then we conduct the data analysis to inspect the impact of HW and OS type along with their currency on the rates of server failures. This can further be used to derive guidelines to support the technology refresh decisions in the data centers.

Today the decision of when to modernize which elements of the server HW/SW stack is often done manually based on simple business rules. In this paper we alleviate this problem by supporting the decision process with an automated approach based on incident tickets and server attributes data. As a first step we identify and rank servers with problematic behavior as candidates for modernization using a random forest classifier. Second, this predictive model is used to evaluate the impact of different modernization actions and suggest the most effective ones. We show that our chosen model yields high quality predictions and outperforms traditional linear regression models on a large set of real data.

Mobile devices are becoming the main entry points to the growing number of cloud applications and services. Unlike traditional approaches, we pursue a flexible architectural model where cloud hosted applications are distributed between mobile devices and the cloud in a bid to improve interaction performance. Given the increasing variety of mobile platforms or virtual instances, in this paper we approach the problem of estimating performance for such applications in two steps. First, we identify the factors that impact interaction response times, such as the application distribution schemes, workload sizes and intensities, or the resource variations of the mobile-cloud setup. Second, we attempt to find correlations between these factors and to understand how to build a unified and generic performance estimation model.