Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

Regulated compliance has become a business liabil-ity given the constant changes to the IT environments driven by the new Cloud normal enabling daily upgrades, and by the new programs related to cybersecurity. To keep pace with the changes, enterprises and auditor agencies shifted from annual audit to continuous compliance. Critical enablers of continuous compliance are standardization and automation. To achieve this, we need to treat everything as code, from the compliance artifacts to the policies and their results. Currently, Compliance as Code (CaC) is utilized to represent compliance artifacts and processes, while Policy as Code (PaC) is employed to express the logic of validating the actual state of the systems against the desired state. Although there is emerging technology for codifying compliance and policy, there are challenges that hinder the compliance digitization transformation towards continuous compliance. The first challenge is how to migrate from document-based operations to the program-based operations. The second challenge arises from the disconnect between CaC and the varied native interfaces of existing PaC solutions. Lastly, there is a concern regarding the reliability of results from end-to-end automation when replacing human-in-the-loop overseeing and guaranteeing the integrity with fully automated process. To address these challenges, we have developed a GitOps-based secure pipeline framework, seamlessly integrating compliance authoring with CaC while ensuring the secure integration of Compliance as Code and Policy as Code, protecting data integrity and preserving data traceability. In this paper, we present the details of the framework which enables end-to-end automation of compliance processes while supporting integration with various Policy Validation Points (PVPs), and showcase the evaluation of our implemented solution's reliability by testing risks such as unintentional modification of policies.

Automation of cybersecurity processes has become crucial with large scale deployment of sensitive workloads in regulated on-prem, private, and public cloud environments. Regulatory and standards bodies such as Payment Card Industry (PCI), Federal Financial Institutions Examination Council (FFIEC), International Organization for Standardization (ISO), and others govern the minimal set of cybersecurity controls that an organization must implement. To meet such requirements while maintaining business agility, organizations need to modernize from manual document based compliance management to automated processes for continuous compliance. This modernized process is called compliance-as-code. In this paper, we present an architecture for compliance-as-code based on a standardized framework. We identify several design choices and our rationale behind those. Specifically, we introduce a system for manipulating compliance information in a standardized manner and a data interchange protocol for inter-operable communication of compliance information. We demonstrate the scalability of our approach and briefly describe deployment and experimental results in real world settings.

Cloud-based analytics allow for inexpensive processing of large amount of data. However, processing protected health information (PHI) in cloud is a challenging task due to strict regulations (e.g., HIPAA) requiring features (e.g., data isolation) which most cloud-based platforms do not currently support in their offerings. This makes it difficult to leverage many technologies well suited to the cloud (e.g., Apache Spark)to process PHI. To address this issue, we have developed the Watson Health Cloud (WHC), a cloud-based platform for the storage and analysis of large amount of PHI. The WHC enables all the features necessary to store and process PHI, with little customization needed by the end-user. This paper describes the lessons learned from developing a cloud platform for PHI. Specifically, we discuss the architecture and implementation challenges we faced throughout development. We hope the insights gained from our experiences help others when designing frameworks and applications which process PHI.

The cloud service marketplace (CSM) is an exploratory project aiming to provide “an AppStore for Services.” It is an intelligent online marketplace that facilitates service discovery and acquisition for enterprise customers. Traditional service discovery and acquisition are time-consuming. In the era of OneClick Checkout and pay-as-you-go service plans, users expect services to be purchased online efficiently and conveniently. However, as services are complex and different from software apps, the currently prevailing App Store based on keyword search is inadequate for services. In CSM, exploring and configuring services are an iterative process. Customers provide their requirements in natural language and interact with the system through questioning and answering. Learning from the input, the system can incrementally clarify users’ intention, narrow down the candidate services, and profile the configuration information for the candidates at the same time. CSM’s back end is built around the Services Knowledge Graph (SKG) and leverages data mining technologies to enable the semantic understanding of customers’ requirements. To quantitatively assess the value of CSM, empirical evaluation on real and synthetic datasets and case studies are given to demonstrate the efficacy and effectiveness of the proposed system.

Infrastructure-as-a-service clouds are becoming widely adopted. However, resource sharing and multi-tenancy have made performance anomalies a top concern for users. Timely debugging those anomalies is paramount for minimizing the performance penalty for users. Unfortunately, this debugging often takes a long time due to the inherent complexity and sharing nature of cloud infrastructures. When an application experiences a performance anomaly, it is important to distinguish between faults with a global impact and faults with a local impact as the diagnosis and recovery steps forfaults with a global impact or local impact are quite different. In this paper, we present PerfCompass, an online performance anomaly fault debugging tool that can quantify whether a production-run performance anomaly has a global impact or local impact. PerfCompass can use this information to suggest the root cause as either an external fault (e.g., environment-based) or an internal fault (e.g., software bugs). Furthermore, PerfCompass can identify top affected system calls to provide useful diagnostic hints for detailed performance debugging. PerfCompass does not require source code or runtime application instrumentation, which makes it practical for production systems. We have tested PerfCompass by running five common open source systems (e.g., Apache, MySQL, Tomcat, Hadoop, Cassandra) inside a virtualized cloud testbed. Our experiments use a range of common infrastructure sharing issues and real software bugs. The results show that PerfCompass accurately classifies 23 out of the 24 tested cases without calibration and achieves 100 percent accuracy with calibration. PerfCompass provides useful diagnosis hints within several minutes and imposes negligible runtime overhead to the production system during normal execution time.

As the cloud services journey through their lifecycle towards becoming commodities, the demand is increasing for "pay-per-use" pricing model. In this model, users are charged for the amount of resources, e.g., Volume of transactions, CPU usage, etc., being consumed during a given time period. Software as a Service (SaaS) providers charging their customers via pay-per-use (e.g., Microsoft Azure Web Services) and facing Infrastructure as a Service (IaaS) costs per VM per month (e.g., Soft Layer) have to carefully choose and scale their non-revenue generating service management infrastructure to penetrate and stay in the market. In this paper, we focus on the metering and rating aspects of cloud service management, and their scalability with the SaaS business and operational changes. We design a framework for cloud service providers to scale their revenue management systems in a cost-aware manner, where the deployment of these revenue systems dynamically uses existing or newly provisioned SaaS VMs, instead of the extant approach of using dedicated setups. Our experimental analysis shows that service management related tasks can be offloaded to the existing VMs with at most 15% overhead in CPU utilization, 10% overhead for memory usage, and negligible overhead for I/O and network usage. We used traces from IBM production servers to mimic the load on VMs. By dynamically scaling the service management setup, we were able to adapt to increasing metering data processing requirements without incurring additional cost, while preserving the infrastructure footprint.

As the cloud services journey through their life-cycle towards commodities, cloud service providers have to carefully choose the metering and rating tools and scale their infrastructure to effectively process the collected metering data. In this paper, we focus on the metering and rating aspects of the revenue management and their adaptability to business and operational changes. We design a framework for IT cloud service providers to scale their revenue systems in a cost-aware manner. The main idea is to dynamically use existing or newly provisioned SaaS VMs, instead of dedicated setups, for deploying the revenue management systems. At on-boarding of new customers, our framework performs off-line analysis to recommend appropriate revenue tools and their scalable distribution by predicting the need for resources based on historical usage. This allows the revenue management to adapt to the ever evolving business context. We evaluated our framework on a test bed of 20 physical machines that were used to deploy 12 VMs within Open Stack environment. Our analysis shows that service management related tasks can be offloaded to the existing VMs with at most 15% overhead in CPU utilization, 10% overhead for memory usage, and negligible overhead for I/O and network usage. By dynamically scaling the setup, we were able to reduce the metering data processing time by many folds without incurring any additional cost.

Service network analysis is an essential aspect of web service discovery, search, mining and recommendation. Many popular web service networks are content-rich in terms of heterogeneous types of entities, attributes and links. A main challenge for ranking services is how to incorporate multiple complex and heterogeneous factors, such as service attributes, relationships between services, relationships between services and service providers or service consumers, into the design of service ranking functions. In this paper, we model services, attributes, and the associated entities, such as providers, consumers, by a heterogeneous service network. We propose a unified neighborhood random walk distance measure, which integrates various types of links and vertex attributes by a local optimal weight assignment. Based on this unified distance measure, a reinforcement algorithm, ServiceRank, is provided to tightly integrate ranking and clustering by mutually and simultaneously enhancing each other such that the performance of both can be improved. An additional clustering matching strategy is proposed to efficiently align clusters from different types of objects. Our extensive evaluation on both synthetic and real service networks demonstrates the effectiveness of ServiceRank in terms of the quality of both clustering and ranking among multiple types of entity, link and attribute similarities in a service network.

For as long as there have been services there has been a desire to have a convenient medium to expose and discover service offerings. Since early on, various efforts have attempted various approaches at the exchange of computational services, prompting the question of whether there is a market for Web services. We believe that a services marketplace should fulfill the promise of an electronic emporium where third party service providers are able to offer their services in a ubiquitous ecosystem, and where service consumers are able to acquire service solutions that are tailored to their requirements. This paper explores the landscape of cloud services marketplaces, where we are, what enablers are needed to realize the vision, and it presents a prospective architecture to that end.

The overwhelming amount of various monitoring and log data generated in multitier IT systems makes problem determination one of the most expensive and labor-intensive tasks in IT Services arena. Particularly the initial step of problem classification is complicated by error propagation making secondary problems surfacing on multiple dependent resources. In this paper, we propose to automate the process of problem classification by leveraging machine learning. The main focus is to categorize the problem a user experiences by recognizing the real root cause specificity leveraging available training data such as monitoring and logs across the systems. We transform the structure of the problem into a hierarchy using an existing taxonomy. We then propose an efficient hierarchical incremental learning algorithm which is capable of adjusting its internal local classifier parameters in realtime. Comparing to the traditional batch learning algorithms, this online solution decreases the computational complexity of the training process by learning from new instances on an incremental fashion. Our approach significantly reduces the memory required to store the training instances. We demonstrate the efficiency of our approach by learning hierarchical problem patterns for several issues occurring in distributed web applications. Experimental results show that our approach substantially outperforms previous methods.

Acquiring a private computing cloud is the first step that an enterprise would choose to enable the cloud model and get its considerable benefits while keeping the control within the enterprise. The enterprise level applications that provide the infrastructure enabling cloud computing services are typically built by integrating inter-related complex software components. Critical challenges of these applications are the increasing level of inter-component dependencies and the customized growth, which make recurrent deployment of such applications, as the one required in private clouds, labor intensive and error prone. In this paper we investigate the type of issues faced when deploying a cloud computing management infrastructure and propose a solution to self-assist the deployment. We show how by leveraging virtual image technologies we can detect faulty installations and their signatures early in the deployment process. We also propose a methodology to capture in a shared repository and update these signatures for reuse in subsequent deployments in the form of two level signature patterns. We explore the perspective of our solution and criteria of analysis.

While IT management services represent a mature subject in the IT business arena, the emerging cloud generation of management services require critical enhancements to the current processes and technologies in order to deliver IT management remotely with rapid on-boarding and minimal labor involvement from experts, to be affordable and scale up to the promise of the cloud. Traditional Remote Infrastructure Management (RIM) service providers use their own Network Operations Centers (NOC) to remotely monitor and manage customerspsila IT infrastructure. The primary business value for RIM services is that it helps global enterprises to small and medium businesses (SMB) to outsource the burden of managing their IT infrastructure. Although the IT management service itself delivered this way is more affordable, the RIM customer on-boarding process particularly is not, taking between one to two months of expensive labor. This paper describes what and how IT management processes, technologies and skills can be improved to provide remote customer on-boarding at an appropriate speed for delivery from the cloud. Our contributions consist of major enhancements in a key on-boarding area, namely IT discovery. Experimental results show that our approach aligns the RIM on-boarding methods to the cloud expectations both from a time as well as quality perspective.

Problem determination and resolution PDR is the process of detecting anomalies in a monitored system, locating the problems responsible for the issue, determining the root cause and fixing the cause of the problem. The cost of PDR represents a substantial part of operational costs, and faster, more effective PDR can contribute to a substantial reduction in system administration costs. In this paper, we propose to automate the process of PDR by leveraging machine learning methods. The main focus is to effectively categorize the problem a user experiences by recognizing the problem specificity leveraging all available training data such like the performance data and the logs data. Specifically, we transform the structure of the problem into a hierarchy which can be determined by existing taxonomy in advance. We then propose an efficient hierarchical incremental learning algorithm which is capable of adjusting its internal local classifier parameters in real-time. Comparing to the traditional batch learning algorithms, this online learning framework can significantly decrease the computational complexity of the training process by learning from new instances on an incremental fashion. In the same time this reduces the amount of memory required to store the training instances. We demonstrate the efficiency of our approach by learning hierarchical problem patterns for several issues occurring in distributed web applications. Experimental results show that our approach substantially outperforms previous methods.

We present SPIRIT, a Service for Providing Infrastructure Recommendations for Information Technology. SPIRIT allows maintenance support providers for Small-to-Medium Businesses (SMBs) to recommend solutions which are standardized (SMBs usually cannot afford customized IT solutions), flexible (accommodating as much as possible the customer’s existing IT environment), and cost-effective (minimizing the cost of upgrading the customer’s environment). SPIRIT works by first aligning the customer’s IT infrastructure with a “template” describing the best practices recommended by the maintenance support provider. Then, the aligned environment can be upgraded by choosing from a standard set of well understood, highly automated (and therefore economical) options. In this paper we present the framework of our solution.

Key areas in managing e-commerce systems are problem prediction, root cause analysis, and automated problem remediation. Anticipating SLO violations by proactive problem determination (PD) is particularly important since it can significantly lower the business impact of application performance problems. The main contribution of this paper is to investigate proactive PD based on two important concepts: dependency graphs and dynamic runtime performance characteristics of resources that comprise an I/T environment. The authors show how one can calculate and use the contribution of all supporting resources for a transaction to the end-to-end SLO for that transaction. Higher order moments of these components' contributions are further tracked for proactive alerting. An important aspect of this process is the classification of user transactions based on the profile of their resource usage, enabling one to set appropriate thresholds for the different classes only. Combined with the complete or semi-complete dependency information, our approach confines the scope of potential root causes to a small set of components, thus enabling efficient performance problem anticipation and quick remediation.

In this paper we propose a novel technique to automatically structure problem tickets consisting of free form, heterogeneous textual data, so that IT problem isolation and resolution can be performed rapidly. The originality of our technique consists in applying the conditional random fields (CRFs) supervised learning process to automatically identify individual units of information in the raw data. The CRFs have been shown to be effective on real-world tasks in various fields. We apply our technique to identify structural patterns specific to the problem ticket data used in call centers to enhance the problem resolution system used by remote technical assistance personnel. Most of the existing ticketing data is not explicitly structured, is highly noisy, and very heterogeneous in content, making it hard to effectively apply common data mining techniques to analyze and search the raw data. An example of such an analysis is the detection of the units of information containing the steps taken by the technical people to resolve a particular customer issue. We present a study of the accuracy of our results.

Managing the service level objectives (SLO) in environments that implement e-commerce systems is a challenging task. It typically involves a clear understanding of how user transactions are supported by the I/T resources that comprise the e-commerce system. This paper investigates a subset of this important management problem. Using transaction to resource dependencies, the authors show how one can experimentally calculate the extent to which supporting resources for a transaction contribute to the end-to-end SLOs for that transaction. An important aspect of this process is the classification of user transactions, based on the profile of their resource usage, enabling one to set appropriate thresholds for different classes. This approach is then used to aid the detection and remediation of application performance bottlenecks.

The growing popularity of e-business has stimulated web sites to evolve from static content servers to complex multi-tier systems built from heterogeneous server platforms. E-businesses now spend a large fraction of their IT budgets maintaining, troubleshooting, and optimizing these web sites. It has been shown that such system management activities may be simplified or automated to various extents if a dynamic dependency graph of the system were available. Currently, all known solutions to the dynamic dependency graph extraction problem are intrusive in nature, i.e. require modifications at application or middleware level. In this paper, we describe non-intrusive techniques based on data mining, which process existing monitoring data generated by server platforms to automatically extract the system component dependency graphs in multi-tier e-business platforms, without any additional application or system modification.