Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

Automated Log Analysis tasks such as root cause analysis and fault prediction play a pivotal role in maintaining the overall application health. These tasks employ log parsers to extract the dynamic (variable) and constant (template) parts of a log line to generate a template. However, our observations indicate that not all templates carry equal significance. Hence, there is a need to prioritize which templates/variables to use for log analysis. In this paper, we introduce LogMId, a Logs-based Metric Identification method, which is designed to extract critical IT metrics from logs. Through LogMId, we aim to en-hance monitoring, observability tools and in turn Site Reliability Engineers to mine better insights from log data. We showcase the effectiveness of LogMId on a popular log analysis task of anomaly detection. Our experiments indicate that integrating previously used benchmark tools with LogMId features lead to improved results. Additionally, LogMId demonstrates effectiveness even with a smaller amount of training data, emphasising its utility.

In cloud-native applications, a large fraction of operational failures, known as outages, result in violations of Service Level Objectives (SLOs). SLOs are defined around specific measurable characteristics: availability, throughput, frequency, response time, and quality. Four metrics, latency, traffic, errors, and saturation, ensure coverage for most outages of an application. These are often called golden signals. The dynamicity and complexity of cloud-native applications complicate Site Reliability Engineers’ (SREs) efforts in problem determination, in particular in its fault localization. The fault localization is often a try-and-error process in which SREs rely on their domain knowledge and experience. It is laborious and frequently results in long Mean Time To Resolution (MTTR) for outages. This paper describes a lightweight fault localization system, that establishes causal relationships among the golden signal service errors and error logs, and further leverages PageRank centrality of the derived causal graph for generating a ranked list of faulty microservices.

Technical support domain involves solving problems from user queries through various channels: voice, web and chat, and is both time-consuming and labour intensive. The textual queries in web or chat mode are unstructured and often incomplete. This affects information retrieval and increases the difficulty level for agents to solve it. Such cases require multiple rounds of interaction between user and agent/chatbot in order to better understand the user query. This paper presents a deployed system called Question Quality Improvement (QQI), that aims to improve the quality of user utterance by understanding and extracting important parts of an utterance and gamifying the user interface, prompting them to enter the remaining relevant information. QQI is guided by an ontology designed for the technical support domain and uses co-reference resolution and deep parsing to understand the sentences. Using the syntactics and semantics in the deep parse tree structure various attributes in the ontology are extracted. The system has been in production for over two years supporting around 800 products resulting in a reduction in the time-to-resolve cases by around 29%, leading to huge cost savings. QQI being a core natural language understanding and metadata extraction technology, directly affects more than 8K tickets everyday. These cases are submitted after 50K edits done on the case based on QQI feedback. QQI outputs are used by other technologies such as search and retrieval, case routing for automated dispatch, case-difficulty-prediction, and by the chatbots supported in each product page.

It is common knowledge in the IT service domain that changes to the system configuration are responsible for a major portion of incidents that result in client outages. However, it is typically very difficult to establish a relationship between changes and incidents as proper documentation takes lower priority at change creation time, as well as during incident management, in order to deal with the tremendous time pressure to quickly implement changes and resolve incidents. As a result, it is often not possible to leverage historical data to perform retrospective analysis to identify any emerging trends linking changes to incidents, or to build predictive models for proactive incident prevention at change creation time. In this paper, we present an approach for establishing causality between changes and incidents through an ensemble of statistics, data classification, and natural language processing techniques. We demonstrate our approach with a real world example.

This paper describes a novel approach for identifying IT outsourcing contract renewal risk ahead of contract expiration by taking into account not only client satisfaction survey results (in the form of numeric scores), but also client interview transcripts (in the form of unstructured text). By using machine learning techniques, the interview transcripts are automatically processed to identify important topics of interest along with an associated sentiment for each topic. The output of the sentiment analysis is then used as an input (in addition to client satisfaction survey scores) to classify contract renewal risk. We show that, by using sentiment analysis to transform unstructured textual information into structured input, the classification accuracy of non-renewing contracts, in particular, is substantially enhanced. Moreover, the topics with negative sentiments can shed light on the root causes of problems leading to contract non-renewal.

Application Programming Interface (API) documents are a typical way of describing legal usage of reusable software libraries, thus facilitating software reuse. However, even with such documents, developers often overlook some documents and build software systems that are inconsistent with the legal usage of those libraries. Existing software verification tools require formal specifications (such as code contracts), and therefore cannot directly verify the legal usage described in natural language text in API documents against code using that library. However, in practice, most libraries do not come with formal specifications, thus hindering tool-based verification. To address this issue, we propose a novel approach to infer formal specifications from natural language text of API documents. Our evaluation results show that our approach achieves an average of 92% precision and 93% recall in identifying sentences that describe code contracts from more than 2500 sentences of API documents. Furthermore, our results show that our approach has an average 83% accuracy in inferring specifications from over 1600 sentences describing code contracts.

Access Control Policies (ACP) specify which principals such as users have access to which resources. Ensuring the correctness and consistency of ACPs is crucial to prevent security vulnerabilities. However, in practice, ACPs are commonly written in Natural Language (NL) and buried in large documents such as requirements documents, not amenable for automated techniques to check for correctness and consistency. It is tedious to manually extract ACPs from these NL documents and validate NL functional requirements such as use cases against ACPs for detecting inconsistencies. To address these issues, we propose an approach, called Text2Policy, to automatically extract ACPs from NL software documents and resource-access information from NL scenario-based functional requirements. We conducted three evaluations on the collected ACP sentences from publicly available sources along with use cases from both open source and proprietary projects. The results show that Text2Policy effectively identifies ACP sentences with the precision of 88.7% and the recall of 89.4%, extracts ACP rules with the accuracy of 86.3%, and extracts action steps with the accuracy of 81.9%.

The modularity and customer centric approach of use cases make them the preferred methods for requirement elicitation, especially in iterative software development processes as in agile programming. Numerous guidelines exist for use case style and content, but enforcing compliance to such guidelines in the industry currently requires specialized training and a strongly managed requirement elicitation process. However, often due to aggressive development schedules, organizations shy away from such extensive processes and end up capturing use cases in an ad-hoc fashion with little guidance. This results in poor quality use cases that are seldom fit for any downstream software activities. We have developed an approach for automated and "edit-time" inspection of use cases based on the construction and analysis of models of use cases. Our models contain linguistic properties of the use case text along with the functional properties of the system under discussion. In this paper, we present a suite of model analysis techniques that leverage such models to validate uses cases simultaneously for their style and content. Such model analysis techniques can be combined with a robust NLP techniques to develop integrated development environments for use case authoring, as we do in Text2Test. When used in an industrial setting, Text2Test resulted in better compliance of use cases, in enhanced productivity and, subsequently, in higher quality of test cases.

Use cases are a key technique to elicit software requirements from the point of view of the user of a system. Their prevalence is noticeable ever since the onset of agile programming techniques. Within SOA projects however, business process models are used for capability analysis and gap detection. Business process models present a global view of the system and hence are more suited for gap detection. Therefore, in practice both these forms of requirements continue to be useful and coexist. Often in big software projects and in distributed development environment such coexisting requirement specifications can grow out of synch. We present here a technique to semi-automatically transform use cases into business processes and to create mapping between them. By preserving the mapping between these forms one can enforce consistency between the two forms of requirements.

Natural language is the preferred form for writing use cases. While a few linguistic techniques exist that extract or validate structured information from unstructured natural language use case, they often cannot be extended beyond their primary language. Extending linguistic analysis and automated validation capabilities across multiple languages is necessary not only for widespread industrial adoption but it helps in analyzing a collection of multilingual use cases (quite frequent in multi-national projects) that need to be aggregated. We have published a UIMA (Unstructured Information Management Architecture) based linguistic engine for analyzing English use cases. In this paper, we report on extension of our linguistic technique to Japanese and effect of such an extension on the automated requirement validation suite.

Web script crashes and malformed dynamically generated webpages are common errors, and they seriously impact the usability of Web applications. Current tools for webpage validation cannot handle the dynamically generated pages that are ubiquitous on today's Internet. We present a dynamic test generation technique for the domain of dynamic Web applications. The technique utilizes both combined concrete and symbolic execution and explicit-state model checking. The technique generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for the PHP programming language. Apollo generates test inputs for a Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 673 faults in six PHP Web applications.

We present 1) a novel linguistic engine made of configurable linguistic components for understanding natural language use case specification; and 2) results of the first of a kind large scale experiment of application of linguistic techniques to industrial use cases. Requirement defects are well known to have adverse effects on dependability of software systems. While formal techniques are often cited as a remedy for specification errors, natural language remains the predominant mode for specifying requirements. Therefore, for dependable system development, a natural language processing technique is required that can translate natural language textual requirements into validation ready computer models. In this paper, we present the implementation details of such a technique and the results of applying a prototype implementation of our technique to 80 industrial and academic use case descriptions. We report on the accuracy and effectiveness of our technique. The results of our experiment are very encouraging.

We present an automatic fault localization technique which leverages time spectra as abstractions for program executions. Time spectra have been traditionally used for performance debugging. By contrast, we use them for functional correctness debugging by identifying pieces of program code that take a "suspicious" amount of time to execute. The approach can be summarized as follows: Time spectra are collected from passing and failing runs, observed behavior models are created using the time spectra collected from passing runs, and deviations from these models in failing runs are identified and scored as potential causes of failures. Our empirical evaluations conducted on three real-life projects suggest that the proposed approach can effectively reduce the space of potential root causes for failures, which can in turn improve the turn around time for fixes.

We present a minimalist approach to model-based test generation. Our approach leverages the information provided in the domain and behavioral models of an application under test to produce a small yet effective set of test cases without additional user input. The domain model consists of UML class diagram with invariants, while the behavioral model consists of UML use cases. Each use case flow has an associated guard condition and a set of updates (to the domain object diagram and the output parameters). We treat the model invariants to enable a novel specify once, test everywhere paradigm. Our approach frees the modeler from the responsibility of specifying appropriate alternate flows on use cases affected by each invariant - our analysis augments the specified use cases with appropriate alternate flows. Our approach then produces suitable testing goals which are refinements of the guard conditions on the augmented flows using a set of fault models. Another salient feature of our approach is generation of verification sequences to ensure that the object diagram updates associated with a given flow are implemented correctly. Our technique uses a novel set of fault models to mutate an object diagram and a novel algorithm which distinguishes between the original and the mutated object diagrams. We describe the techniques used in our test generation approach.

In this paper, we describe results of a case study to establish the feasibility of deriving mappings between an abstract user level specification and the code elements in a concrete implementation of a highly secure smart card operating system. Such a mapping is necessary for feedback-directed specification-based test generation to improve code coverage, needed by the stringent criteria for high-assurance systems. We used test cases generated from the user level specification to identify the executed code elements and attempted to use static analysis to map the unexecuted code elements to the corresponding elements in the user level specification. Our primary result is evidence that, given a sufficiently expressive user level specification and a test generation system that is able to effectively use such a specification, the resulting tests will cover the vast majority of the code branches that are able to be covered. Therefore, the benefit of a feedback-directed system will be limited. We further provide evidence that the static analysis required to generate feedback in these cases tends to be difficult, involving inferring the semantics of the internal implementation of data structures. In particular, we observed that the internal states at the implementation level in a high security application pose significant challenges to this mapping process.

Web script crashes and malformed dynamically-generated Web pages are common errors, and they seriously impact usability of Web applications. Current tools for Web-page validation cannot handle the dynamically-generated pages that are ubiquitous on today's Internet. In this work, we apply a dynamic test generation technique, based on combined concrete and symbolic execution, to the domain of dynamic Web applications. The technique generates tests automatically, uses the tests to detect failures, and minimizes the conditions on the inputs exposing each failure, so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for PHP. Apollo generates test inputs for the Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 214 faults in 4 PHP Web applications.

We present an automated approach to generate functional conformance tests for semantic Web services. The semantics of the Web services are defined using the inputs, outputs, preconditions, effects (IOPEs) paradigm. For each Web service, our approach produces testing goals which are refinements of the Web service preconditions using a set of fault models. A novel planner component accepts these testing goals, along with an initial state of the world and the Web service definitions to generate a sequence of Web service invocations as a test case. Another salient feature of our approach is generation of verification sequences to ensure that the changes to the world produced by an effect are implemented correctly. Lastly, a given application incorporating a set of semantic Web services may be accessible through several interfaces such as 1) direct invocation of the Web services, or 2) a graphical user interface (GUI). Our technique allows generation of executable test cases which can be applied through both interfaces. We describe the techniques used in our test generation approach. We also present results which compare two approaches: an existing manual approach without the formal IOPEs information and the IOPEs-based approach reported in this paper. These results indicate that the approach described here leads to substantial savings in effort with comparable results for requirements coverage and fault detection effectiveness.

In this paper, we describe the contributions to the workshop on domain specific approaches to software test automation. The workshop consists of six accepted papers and a keynote speech. The papers tackle a wide range of topics related to the testing of applications within specific domains including event driven software, synchronous safety critical software, web applications, wireless sensor networks, and mobile phone applications. Each of the papers describes the influence of the domain on the testing process.

The growing acceptance of model-based test generation in industry has created a need for a tester-friendly model construction process and associated tools. In this paper, we present an automated approach for checking consistency and completeness of testing models for interactive systems. The testing models are described in terms of operations provided by the system under test (SUT). Each operation is specified as a set of possible results each with a guard condition and a set of actions on its parameters and the system state. The consistency and completeness checks performed include identification of malformed models, irrelevant state variables, incomplete and redundant guard conditions, missing data values, and nondeterminism in and executability of operation results. We review our testing requirements model, and describe various consistency checks performed on this model. We illustrate these checks with an example test model, and also present algorithms used to perform our analysis

We present a framework and tool for porting assessment and code remediation in commercial porting projects, primarily targeting interplatform C/C++ porting. Our framework has a modular architecture for handling real-world code bases, a key part of which is an extensible knowledge base of porting issue types. Porting issue detection and remediation are achieved through dynamically-linked detector and remediation "plug-ins" which can be augmented over time. Analysis and remediation can be performed over multiple intermediate forms, such as the AST or error logs. Analysis and remediation results can be presented as a single assessment report, or the changes can be incorporated into the original code base. Furthermore our framework has been integrated into Eclipse, providing a user interface and allowing further capability for integration with other Eclipse extensions. A subset of our tool has been prototyped and benchmarked against real-world customer code.

Model-based test generation (MBTG) is becoming an area of active research. Several MBTG approaches extract a finite state machine (FSM) from a given model, and use structural (mostly transition) coverage of the extracted FSM as a test selection criteria. In this paper, we demonstrate inadequacy of structural coverage criteria, and propose a set of test selection criteria for extracted FSMs. Our models are described in terms of operations provided by the system under test (SUT). Each operation is specified as a set of possible results each with a guard condition and a set of update actions on its parameters and the system state. The proposed test selection criteria are based on (1) mutations of guard conditions and update actions, (2) concept of a session, which targets errors of SUT not committing the updated system state to persistent storage, and (3) 2-way coverage of independent operations available in a given FSM state. We describe an AI planning based algorithm for finding a sequence of operation invocations to satisfy our proposed test selection criteria. We illustrate our test selection criteria, and report results of a case study which compares fault detection capability of our proposed test selection criteria with that of structural criteria.

Automation of test design during the function testing phase is essential both to reduce the substantial cost of testing and to improve the delivered software reliability. We argue for a model based approach specifically designed from a test perspective to automate test design. We describe features of the Specification and Abstraction Language for Testing (SALT) environment which embodies this perspective. SALT allows testers to capture relationships among partitions of input and output variables for a function under test. The tester can also specify (potential) updates to context which result from the function invocation. This context enables the generation of sequences of function invocations with expected outputs. These test specifications along with a fault model, allow generation of an optimized set of test variations. We describe an example to illustrate SALT usage and report results of our pilot study using SALT.

Multi-panel systems are systems that interact with a user via multiple input panels. The flow through the panels is influenced by the interaction. Multi-panel systems are ubiquitous, and include panel-based legacy applications, automated teller machines, and Web-based systems. Finding regression test suites which efficiently cover the functionality of these systems is difficult, because it requires covering interactions between input fields within a panel as well as flows between panels. Previous approaches to covering input field interactions include partitioning methods and combinatorial design methods. State machines and decision tables have been used to cover flow between panels. None of these techniques produce a conceptually simple, unified model that supports both intra- and inter-panel coverage. Our new method for capturing and representing test specifications provides such a model. This model is then used to generate a locally minimal set of test cases which completely covers the model. We applied this technique in a pilot study for regression testing, and this pilot had promising results which we discuss. We conclude by presenting our plans for generalizing the technique beyond multi-panel systems and regression testing.

We propose a new technique for automatic generation of test cases for predicates. Earlier we proposed an efficient and effective test generation strategy for Boolean expressions. We now extend this strategy to predicates. Our new strategy addresses a number of issues, including: analysis of dependencies between relational expressions in a predicate P; generation of test constraints for P based on the detection of Boolean and relational operator faults in P; and generation of actual tests according to the generated test constraints for P. We propose the use of constraint logic programming (CLP) to automate test data generation for a predicate. Furthermore, we propose an incremental approach to apply CLP techniques to solve a constraint system. Since our technique is specification-based, it can facilitate generation of expected outputs for actual tests.

The authors propose a new technique for the automatic generation of test cases for predicates. Earlier, they proposed an efficient effective test generation strategy for Boolean expressions. They now extend this strategy to predicates. Their new strategy addresses several issues, including: analysis of dependencies between relational expressions in a predicate /spl Pscr/; generation of test constraints for /spl Pscr/ based on the detection of Boolean and relational operator faults in /spl Pscr/; and generation of actual tests according to the generated test constraints for /spl Pscr/. They propose: the use of constraint logic programming (CLP) to automate test-data generation for a predicate; and an incremental approach to apply CLP techniques to solve a constraint system. Since their technique is specification-based, it can facilitate generation of anticipated outputs for actual tests.

We propose a new strategy for generating test cases for Boolean expressions. In the past, we reported the BOR (Boolean Operator) strategy for generating test cases for predicates which are singular: which contain only one occurrence of each constituent Boolean variable. We also reported results of the empirical studies that were carried out to study the effectiveness of the strategy, but the BOR algorithm did not work well with non-singularities: multiple occurrences of constituent Boolean variables. The solution we propose for the problem is a combination of the original BOR strategy and the MI (Meaning Impact) strategy reported elsewhere. Our approach is to divide a Boolean expression into components that do not have common variables, apply the MI strategy to non-singular components, and the BOR strategy to singular components, and then apply the BOR strategy to combine the test sets generated for all component. Our empirical results indicate that our hybrid approach produces fewer tests for a Boolean expression than the MI strategy. The fault detection capability of our proposed approach has also been found to be comparable to that of the MI strategy. Our test generation strategy can be used to improve the reliability and safety of a program.

We report the results of three empirical studies of fault detection and stability performance of the predicate-based BOR (Boolean Operator) testing strategy. BOR testing is used to develop test cases based on formal software specification, or based on the implementation code. We evaluated the BOR strategy with respect to some other strategies by using Boolean expressions and actual software. We applied it to software specification cause-effect graphs of a safety-related real-time control system, and to a set of N-version programs. We found that BOR testing is very effective at detecting faults in predicates, and that BOR-based approach has consistently better fault detection performance than branch testing, thorough (but informal) functional testing, simple state-based testing, and random testing. Our results indicate that BOR test selection strategy is practical and effective for detection of faulty predicates and is suitable for generation of safety-sensitive test-cases.>

The timing performance of N-version multi-stage software is analyzed for a strategy called expedient voting. In expedient voting the voting takes place as soon as an adequate number of components have finished the stage. The concept of a 'runahead' is introduced: the faster versions are allowed to run ahead of the rest of the slower versions by one or more stages, with synchronized re-start in the event of a failure. If the versions are highly reliable, inter-version failure dependence is small, and the difference between the fastest and the slowest successful components in each stage is large, then the execution speed-up through expedient voting may be substantial. Runaheads exceeding three stages offer diminishing returns. Speed-up deteriorates with reduction in the version reliability and independence.>