In the rapidly evolving landscape of site reliability engineering (SRE), the demand for efficient and effective solutions to manage and resolve issues in site and cloud applications is paramount. This paper presents an innovative approach to action automation using large language models (LLMs) for script generation, assessment, and refinement. By leveraging the capabilities of LLMs, we aim to significantly reduce the human effort involved in writing and debugging scripts, thereby enhancing the productivity of SRE teams. Our experiments focus on Bash scripts, a commonly used tool in SRE, and involve the CodeSift dataset of 100 tasks and the InterCode dataset of 153 tasks. The results show that LLMs can automatically assess and refine scripts efficiently, reducing the need for script validation in an execution environment. Results demonstrate that the framework shows an overall improvement of 7-10% in script generation.

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

Enterprise organizations have large amounts of data which is utilized by multiple Machine Learning (ML) models over various software frameworks. These models provide trends and insights from the data that can help enterprises define business rules around their processes. However, if certain aspects of this data are removed from the datasets, it could influence the business rules and policies in place. When a user requests data to be removed, the model retraining may be required called Machine Unlearning (MU). Recent research works in the area of MU include different methods of retraining the machine learning models. It turns out that there is lack of work in removing certain aspects of data, and quantifying its impact on the models. This paper aspires to provide a novel methodology IDMU (Impact Driven Machine Unlearning) that performs quantification of the impact of data removal requests while performing MU. Our method provides recommendations for data removal requests, factoring in underlying features of data. The results from the industrial application and evaluation of our method on a financial services dataset are encouraging. The overall IDMU had a mean MAPE of 10.25% over a set of 120 data removal requests. It also saved ~1900 hours of model retraining time by factoring in urgency and impact of data removal requests over a period of three years.

Enterprise organizations have vast datasets that need comprehensive analysis on a frequent basis, in order to manage data and take business decisions based on it. However, we observe that there can be a lack of industry standards for definitions of key terms. Additionally, there is a lack of governance for maintaining business processes. This typically leads to disconnected siloed datasets generated from disintegrated systems. To address these challenges, we developed a novel, integrated methodology DaME (Data Mapping Engine) that performs data mapping using ensemble of NLP techniques.The results from the industrial application and evaluation of DaME on a financial services dataset are encouraging that it can help reduce manual effort by automating data mapping and reusing the learning. The accuracy from our dataset in the application is much higher at 69% compared to the existing state-of-the-art with an accuracy of 34%. It has also helped improve the productivity of the industry practitioners, by saving them 14,000 hours of time spent manually mapping vast data stores over a period of ten months.

For a cloud service provider, the goal is to proactively identify signals that can help reduce outages and/or reduce the mean-time-to-detect and mean-time-to-resolve. After an incident is reported, the Site Reliability Engineers diagnose the fault and search for a resolution by formulating a textual query to find similar historical incidents - this approach is called text-based retrieval. However, it has been observed that the formulated queries are inadequate and short. An alternate approach, presented in this paper, integrates information spread across heterogeneous and siloed datasets, as a ready-to-use knowledge base for metadata-based resolution retrieval. Additionally, it exploits historical problem context for building metadata prediction models which are used at run-time for automatically formulating queries from log anomalies detected by the Log Anomaly Detection module. The query, thus formed, is run against the metadata-based index, unlike the text-based index in text retrieval, resulting in superior performance, in terms of relevancy of the resolution documents retrieved. Through experiments on web application server applications deployed on the cloud, we show the efficacy of metadata-based retrieval, which not only returns targeted results as compared to text-based retrieval but also the relevant resolution document appear amongst the top 3 positions for 60% of the queries.

Based on 2020 SRE report, 80% of SREs work on postmortem analysis of incidents due to lack of provided information and 16% of toil come from investigating false positives/negatives. As a cloud service provider, the desire is to proactively identify signals that can help reduce outages and/or reduce the mean time to resolution. By leveraging AI for Operations (AIOps), this work proposes a novel methodology for proactive identification of log anomalies and its resolutions by sifting through the log lines. Typically, relevant information to retrieve resolutions corresponding to logs is spread across multiple heterogeneous corpora that exist in silos, namely historical ticket data, historical log data, and symptom resolution available in product documentation, for example. In this paper, we focus on augmented dataset preparation from multiple heterogeneous corpora, metadata selection and prediction, and finally, using these elements during run-time to retrieve contextual resolutions for signals triggered via logs. For early evaluation, we used logs from a production middleware application server, predicted log anomalies and their resolutions, and conducted qualitative evaluation with subject matter experts; the accuracy of metadata prediction and resolution retrieval are 78.57% and 65.7%, respectively.

We describe disaster protection and recovery of cloud-hosted enterprise applications both at the cloud infrastructure level and at the application level. We explore scenarios which favor one option over the other, and scenarios where a combination of both are required for effective and end-to-end protection. Through case studies grounded in the experience of implementing disaster recovery for IBM's Cloud Managed Services (CMS) platform, we highlight the complexities of protecting enterprise applications on the cloud. For recovery planning and execution, we present a scheduling algorithm that recovers machines hosted on cloud by taking into account application-level logical dependencies and the business criticalities of the applications.

Cloud environments offer flexibility, elasticity, and low cost compute infrastructure. Enterprise-level workloads â?? such as SAP and Oracle workloads - require infrastructure with high availability, clustering, or physical server appliances. These features are often not part of a typical cloud offering, and as a result, businesses are forced to run enterprise workloads in their legacy environments. To enable enterprise customers to use these workloads in a cloud, we enabled a large number of SAP and Oracle workloads in the IBM Cloud Managed Services (CMS) for both virtualized and non-virtualized cloud environments. In this paper, we discuss the challenges in enabling enterprise class applications in the cloud based on our experience on providing a diverse set of platforms implemented in the IBM CMS offering.

Enterprise-level workloads - such as SAP and Oracle workloads - require infrastructure with high availability, clustering, or physical server appliances, features which are often not a part of a cloud offering. As a result, businesses are forced to run enterprise workloads in their legacy environments, and cannot take advantage of the cloud's flexibility, elasticity, and low cost. IBM Cloud Managed Services (CMS) cloud implements shared storage, clustering support, and private networks. These features effectively enable a large number of SAP and Oracle workloads to run in both virtualized and non-virtualized cloud environments. In this paper, we discuss a diverse set of enterprise applications implemented in the IBM CMS cloud.

Due to increased competition in the IT Services business, improving quality, reducing costs and shortening schedules has become extremely important. A key strategy being adopted for achieving these goals is the use of an asset-based approach to service delivery, where standard reusable components developed by domain experts are minimally modified for each customer instead of creating custom solutions. One example of this approach is the use of contract templates, one for each type of service offered. A compliance checking system that measures how well actual contracts adhere to standard templates is critical for ensuring the success of such an approach. This paper describes the use of document similarity measures - Cosine similarity and Latent Semantic Indexing - to identify the top candidate templates on which a more detailed (and expensive) compliance analysis can be performed. Comparison of results of using the different methods are presented.

Key areas in managing e-commerce systems are problem prediction, root cause analysis, and automated problem remediation. Anticipating SLO violations by proactive problem determination (PD) is particularly important since it can significantly lower the business impact of application performance problems. The main contribution of this paper is to investigate proactive PD based on two important concepts: dependency graphs and dynamic runtime performance characteristics of resources that comprise an I/T environment. The authors show how one can calculate and use the contribution of all supporting resources for a transaction to the end-to-end SLO for that transaction. Higher order moments of these components' contributions are further tracked for proactive alerting. An important aspect of this process is the classification of user transactions based on the profile of their resource usage, enabling one to set appropriate thresholds for the different classes only. Combined with the complete or semi-complete dependency information, our approach confines the scope of potential root causes to a small set of components, thus enabling efficient performance problem anticipation and quick remediation.

In this paper we propose a novel technique to automatically structure problem tickets consisting of free form, heterogeneous textual data, so that IT problem isolation and resolution can be performed rapidly. The originality of our technique consists in applying the conditional random fields (CRFs) supervised learning process to automatically identify individual units of information in the raw data. The CRFs have been shown to be effective on real-world tasks in various fields. We apply our technique to identify structural patterns specific to the problem ticket data used in call centers to enhance the problem resolution system used by remote technical assistance personnel. Most of the existing ticketing data is not explicitly structured, is highly noisy, and very heterogeneous in content, making it hard to effectively apply common data mining techniques to analyze and search the raw data. An example of such an analysis is the detection of the units of information containing the steps taken by the technical people to resolve a particular customer issue. We present a study of the accuracy of our results.