In the rapidly evolving landscape of site reliability engineering (SRE), the demand for efficient and effective solutions to manage and resolve issues in site and cloud applications is paramount. This paper presents an innovative approach to action automation using large language models (LLMs) for script generation, assessment, and refinement. By leveraging the capabilities of LLMs, we aim to significantly reduce the human effort involved in writing and debugging scripts, thereby enhancing the productivity of SRE teams. Our experiments focus on Bash scripts, a commonly used tool in SRE, and involve the CodeSift dataset of 100 tasks and the InterCode dataset of 153 tasks. The results show that LLMs can automatically assess and refine scripts efficiently, reducing the need for script validation in an execution environment. Results demonstrate that the framework shows an overall improvement of 7-10% in script generation.

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

An incident which is created due to a fault in an Application Monitoring System, gather large amount of diverse information, which helps in effective remediation of the fault. A Site Reliability Engineer (SRE) should resolve the outage quickly, for which all the fault related information should be presented to her in a crisp and summarized form. In this paper, we address this problem by summarizing an incident and presenting important details to the SRE. We group the list of related events, which is a part of the incident payload, and use Large Language Models (LLMs) to summarize each of these groups separately. The grouping is driven by the entities and symptoms occurring due to the fault, and used to design efficient prompts for LLM. Our approach addresses the known issue of LLM hallucination, and remove any false symptoms or facts from the generated summary. Our proposed method creates a resource-entity driven summarization, giving a bird's eye view of the entire outage in a cost and time efficient way, thus aiding an SRE to understand and resolve the incident at a faster pace.

AI has been extensively used to help Site Reliability Engineers (SREs) to resolve faults in cloud services and applications. It helps to accelerate resolution time by navigating through the vast amount of heterogeneous data (logs, metrics, alerts, etc) related to a fault. A good ITOps system should help SREs by giving precise and meaningful insights for a quick understanding of the data at hand. In this paper, we design a framework to summarize the context or insight present in the heterogeneous data related to a fault. The proposed framework constructs queries/prompts, specific to the ITOps domain, which helps us to generate more insightful abstractive summaries using state-of-the-art text generator models. Initial study on simulated faults shows promising results, which can be expanded to accommodate other datatype, providing summaries for real-world cases.

In many real-world applications, labeled training data is hard to come by for text classification. These tasks are often domain specific, where the vocabulary of the textual input is different than that of the general language vocabulary. In this paper, we deal with one of such tasks of automation of a software monitoring system, where logs are analyzed in real-time. We describe a weakly supervised method to process incoming streams of logs for identifying fault types in logs. We propose hand-crafted feature extractions, specially designed for the classifiers for log inputs. In order to make the processing time efficient and generalizable across various log sources, we rely on a weak supervised fault classifier, where the domain knowledge is incorporated using a word embedding mode built on a domain specific corpus. Experiments on logs obtained from various applications show the efficacy of our proposed method.

As an increasing number of organizations migrate to the cloud, the main challenge before an operations team is how to effectively use an overwhelming amount of information derivable from multiple data sources like logs, metrics, and traces to help maintain the robustness and availability of cloud services. Site Reliability Engineers (SRE) depend on periodic log data to understand the state of an application and to diagnose the potential root cause of a problem. Despite best practices, service outages happen and result in the loss of billions of dollars in revenue. Many a times, indicators of these outages are buried in the flood of alerts which an SRE receives. Therefore, it is important to reduce noisy alerts so that an SRE can focus on what is critical. Log Anomaly Detection detects anomalous system behaviours and finds patterns (anomalies) in data that do not conform to expected behaviour. Different anomaly detection techniques have been incorporated into various AIOps platforms, but they all suffer from a large number of false positives. Also, some anomalies are transient and resolve on their own. In this paper, we propose an unsupervised model-agnostic persistent anomaly detector based on golden signal based signatures, as a post-processing filtering step on detected anomalies, so we don’t have to interfere with the existing deployed anomaly detector in a system.

Conversational channels are changing the landscape of hybrid cloud service management. These channels are becoming important avenues for Site Reliability Engineers (SREs) %Subject Matter Experts (SME) to collaboratively work together to resolve an incident or issue. Identifying segmented conversations and extracting key insights or artefacts from them can help engineers to improve the efficiency of the incident remediation process by using information retrieval mechanisms for similar incidents. However, it has been empirically observed that due to the semi-formal behavior of such conversations (human language) the conversations are very unique in nature and also contain domain-specific terms. %It is important to identify the correct keywords and artefacts like symptoms, issue etc., present in the conversation chats. In this paper, we build a framework that taps into the conversational channels and uses various learning methods to (1) understand and extract key artefacts from conversations like diagnostic steps and resolution actions taken and (2) present an approach to identify past conversations about similar issues. Experimental results on our dataset show the efficacy of the methods used in our proposed system.

A novel technique, called augmented whole-body scanning via magnifying PET (AWSM-PET), that improves the sensitivity and lesion detectability of a PET scanner for whole-body imaging is proposed and evaluated. A Siemens Biograph Vision PET/CT scanner equipped with one or two high-resolution panel-detectors was simulated to study the effectiveness of AWSM-PET technology. The detector panels are located immediately outside the scanner's axial field-of-view (FOV). A detector panel contains 2 × 8 detector modules each consisting of 32 × 64 LSO crystals (1.0 × 1.0 × 10.0 mm3each). A22Na point source was stepped across the scanner's FOV axially to measure sensitivity profiles at different locations. An elliptical torso phantom containing 7 × 9 spherical lesions was imaged at different axial locations to mimic a multi-bed-position whole-body imaging protocol. Receiver operating characteristic (ROC) curves were analyzed to evaluate the improvement in lesion detectability by the AWSM-PET technology. Experimental validation was conducted using an existing flat-panel detector integrated with a Siemens Biograph 40 PET/CT scanner to image a torso phantom containing spherical lesions with diameters ranging from 3.3 to 11.4 mm. The contrast-recovery-coefficient (CRC) of the lesions was evaluated for the scanner with or without the AWSM-PET technology. Monte Carlo simulation shows 36%-42% improvement in system sensitivity by a dual-panel AWSM-PET device. The area under the ROC curve is 0.962 by a native scanner for the detection of 4 mm diameter lesions with 5:1 tumor-to-background activity concentration. It was improved to 0.977 and 0.991 with a single- and dual-panel AWSM-PET system, respectively. Experimental studies showed that the average CRC of 3.3 mm and 4.3 mm diameter tumors were improved from 2.8% and 4.2% to 7.9% and 11.0%, respectively, by a single-panel AWSM-PET device. With a high-sensitivity dual-panel device, the corresponding CRC can be further improved to 11.0% and 15.9%, respectively. The principle of the AWSM-PET technology has been developed and validated. Enhanced system sensitivity, CRC and tumor detectability were demonstrated by Monte Carlo simulations and imaging experiments. This technology may offer a cost-effective path to realize high-resolution whole-body PET imaging clinically.

Recently, deep learning frameworks have been shown to learn a feature embedding that captures fine-grained image similarity using image triplets or quadruplets that consider pairwise relationships between image pairs. In real-world datasets, a class contains fine-grained categorization that exhibits within-class variability. In such a scenario, these frameworks fail to learn the relative ordering between - (i) samples belonging to the same category, (ii) samples from a different category within a class and (iii) samples belonging to a different class. In this paper, we propose the quadlet loss function, that learns an order-preserving fine-grained image similarity by learning through quadlets (query:q, positive:p, intermediate:i, negative:n) where p is sampled from the same category as q, i belongs to a fine-grained category within the class of q and n is sampled from a different class than that of q. We propose a deep quadlet network to learn the feature embedding using the quadlet loss function. We present an extensive evaluation of our proposed ranking model against state-of-the-art baselines on three datasets with fine-grained categorization. The results show significant improvement over the baselines for both order-preserving fine-grained ranking task and general image ranking task.

Domain adaptation is used for machine learning tasks, when the distribution of the training (obtained from source domain) set differs from that of the testing (referred as target domain) set. In the work presented in this study, the problem of unsupervised domain adaptation is solved using a novel optimisation function to minimise the global and local discrepancies between the transformed source and the target domains. The dissimilarity in data distributions is the major contributor to the global discrepancy between the two domains. The authors propose two techniques to preserve the local structural information of source domain: (i) identify closest pair of instances in source domain and minimise the distances between these pairs of instances after transformation; (ii) preserve the naturally occurring clusters present in source domain during transformation. This cost function and constraints yield a non‐linear optimisation problem, used to estimate the weight matrix. An iterative framework solves the optimisation problem, providing a sub‐optimal solution. Next, using orthogonality constraint, an optimisation task is formulated in the Stiefel manifold. Performance analysis using real‐world datasets show that the proposed methods perform better than a few recently published state‐of‐the‐art methods.

This study describes a new technique of unsupervised domain adaptation based on eigenanalysis in kernel space, for the purpose of categorisation tasks. The authors propose a transformation of data in source domain, such that the eigenvectors and eigenvalues of the transformed source domain become similar to that of the target domain. They extend this idea to the reproducing kernel Hilbert space, which enables to deal with non‐linear transformation of source domain. They also propose a measure to obtain the appropriate number of eigenvectors needed for transformation. Results on object, video and text categorisations tasks using real‐world datasets show that the proposed method produces better results when compared with a few recent state‐of‐art methods of domain adaptation.

This paper describes a method of cross-domain object and event categorization, using the concept of domain adaptation. Here, a classifier is trained using samples from the source/ auxiliary domain and performance is observed on a set of test samples taken from a different domain, termed as the target domain. To overcome the difference between the two domains, we aim to find an optimal sub-space such that the instances from both the domains follow similar distributions when projected onto the sub-space. Along with the distributions, the underlying manifolds of the two domains are aligned in the sub-space to reduce the difference in structure of the data from the two domains. The local spatial arrangement of the instances in both the domains are also preserved in the optimal sub-space. Results show that the proposed method of unsupervised domain adaptation provides better classification accuracy than a few state of the art methods.

This paper describes a method of unsupervised color texture segmentation by efficiently combining different features obtained from multi-channel and multi-resolution filters. The DWT and DCT features are extracted separately from 3 color bands of the image and then fused together for optimal performance. The features are then ranked according to a selection criteria. We propose a new correlation measure for the task of feature ranking. To select the best combination of features to be used, we use the property of cluster scatter of a selected set of features. Finally, the optimum number of ranked order features are used for segmentation using a fuzzy C-Means classifier. The performance of the proposed segmentation method is verified using standard benchmark datasets.