Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the CLASSES2TEST dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.

Social media have become a global phenomenon affecting people in their private lives and in their personal interactions, particularly among younger people. It is thus not surprising that social media are also being explored in professional contexts such as in enterprises, where a number of social media platforms and social extensions to existing workgroup and collaboration systems have been emerging. In this paper, we consider one such platform that was developed for the internal use in a large global enterprise (HP). We present data and analysis of how this social media platform has been used in HP over the past five years. We then present conclusions from this analysis and relate them to work patterns in enterprises with the goal of advancing social media to the next level making them better fit the work context and more relevant for people in their work functions. We consider enterprise sales processes as a case study and present a number of extensions for our social media platform.

As IT services grow more and more complicated, and their management becomes increasingly challenging, IT support organizations assume an essential role to ensure the delivery of Service Level Objectives. What-if scenario analysis represents a very effective tool for the performance optimization of IT support organizations, as it enables an iterative and customized performance optimization process. The problem of accurately modeling IT support organizations requires the development of sophisticated models as well as dedicated parameter inference techniques and tools. This paper presents a multiple-priority queuing model suited for the reenactment of IT support groups developed from the analysis of empirical evidence, as well as a powerful method to infer the model parameters. We applied our model to reenact the behavior of a real life IT support group with our Symian simulator. The results demonstrate that multiple-priority queuing models can reproduce real life IT support groups with a high degree of accuracy.

IT services are getting increasingly complicated, and require IT support organization to manage them. IT support organization are in charge of the incident management process and represent mission-critical structures whose performance needs to be frequently assessed and optimized. State-of-the-art research in the performance optimization of IT support organization proposes user-driven performance assessment and optimization processes based on what-if scenario analysis tools that implement sophisticated IT support organization models. This manuscript instead represents a preliminary study of a different kind of optimization processes, of the closed-loop type, that try to autonomously identify optimal IT support organization configurations according to inputs provided by the user. This paper discusses the development challenges in realizing decision support tools for closed-loop optimization processes and presents a prototype system. The preliminary evaluation of our tool demonstrates that closed-loop processes might be impractical as reference tools but can effectively complement and extend human-driven ones.

IT support organizations are in charge of restoring normal operations after IT service disruptions, among other tasks. Such organizations can be complex systems, with a large network of interacting support groups subject to complex management policies. The performance assessment and optimization of IT support organizations is an extremely challenging task that requires considering organization-specific structure, behavior, and business-level objectives. This paper presents Symian-Web, a decision support tool that enables IT managers to both assess and improve IT support organization performance by using what-if scenario analysis. Symian-Web features advanced information visualization concepts and metaphors, allowing for precise and timely assessment of IT support organization performance, and facilitating their redesign. Symian-Web is realized as a cloud computing-based Web application, thus enabling the tool to take advantage of the on-demand computational capabilities provided by the cloud.

Business-driven IT management (BDIM) aims at ensuring successful alignment of business and IT through thorough understanding of the impact of IT on business processes and business results, and vice versa. This thesis reviews the state of the art of BDIM research and advances it by contributing a comprehensive BDIM solution for the process of IT incident management. The solution can be used as a template for applying the BDIM methodology to other IT service management processes. The work presented in this dissertation resulted in three patent applications and is at the core of the HP IT Analytic™ product (formerly HP DecisionCenter™). This thesis was defended on March 2009.

Business-driven IT management (BDIM) is a recent research effort to drive IT management decisions from a business perspective by considering business indicators such as profit, cost, and customer experience. BDIM studies complicated decision making processes, dealing with the relationship between the IT function and the business value it generates. The present paper aims at stimulating the discussion on decision making theory and practice within the BDIM research community, in order to develop a better understanding of the theoretical background and consequently improve current tools and practices. To this end, the paper analyzes the most challenging aspects of decision making in BDIM and proposes a few discussion topics that could be of interest for future research studies.

There is an intuitive notion that the costs associated with project support actions, currently deemed too high and increasing, are directly related to the effort spent during their development and test phases. Despite the importance of systematically characterizing and understanding this relationship, little has been done in this realm mainly due to the lack of proper tooling for both sharing information between IT project phases and learning from past experiences. To tackle this issue, in this paper we propose a solution that, leveraging existing IT project lifecycle data, is able to predict support costs. The solution has been evaluated through a case study based on the ISBSG dataset, producing correct estimates for more than 80% of the assessed scenarios.

The reuse of knowledge acquired by operators to diagnose failures in Information Technology (IT) infrastructures has potential to decrease the recurrence of failures and, consequently, reduce possible losses and maintenance costs. Nevertheless, existing solutions to support failure diagnosis lack of flexibility to adapt to a constantly changing IT environment. As a result, diagnostic is performed in an ad hoc and static fashion, which hampers the reuse of knowledge to solve similar failures affecting different elements of an IT infrastructure. To bridge this gap, in this paper we propose an extension of Common Information Model (CIM), supported by a conceptual solution for the identification of the root causes of problems, adaptable to changes in the target infrastructure and applicable to similar failures. Experiments carried out considering typical failures during the deployment of IT changes provide evidence about the efficacy of the proposed solution.

We explore the challenge of selecting the best among a set of alternative IT investments. Solving this problem is important since the difference between alternative investment options may be drastic in terms of business results, both positive and negative. The resulting model takes as input a set of investment alternatives and a parameterized description of IT services, and provides as output a Preference Index for each alternative. The solution takes into account such characteristics as epistemic and aleatory uncertainty as well as the decision maker's attitude toward risk. Through a case study and a sensitivity analysis, we conclude that the model is useful in practice and robust; we also describe its domain of validity.

In a large and complex outsourcing organization, with millions of incidents being handled by hundreds of geographically dispersed workgroups, each workgroup is typically measured on the number of incidents it receives, the percentage it resolves, and the average duration for each incident it processes. Incidents that are misrouted will take longer to be resolved. This can lead to the situation where all workgroups are meeting their individual service levels, but the overall service still delivers poor customer satisfaction. In this paper we describe a new approach to analyzing the efficiency of an IT service organization using existing data to provide insight into the quality of incident transfer between workgroups. Based on this approach, we have developed a tool - the IT X-ray - that an analyst can use to quickly provide service organizations with an indication of where incident routing problems are occurring, leading to increased customer satisfaction and reduced delivery costs.

The proper management of IT infrastructures is essential for organizations that aim to deliver high quality services. Given the dynamics of these infrastructures, changes become imminent. In some cases, these changes might raise failures, causing disruption to provided services and consequently affecting the business continuity. Therefore, it is strongly recommended to evaluate the risks associated with changes before their actual execution. Learning from information of past deployed changes it is possible to estimate the risks for recently planned ones. Thereby, in this paper, we propose a solution to weigh the information available from past executed plans by the similarity calculated in relation with the analyzed change plan. A prototype system has been developed in order to evaluate the effectiveness of the solution over an emulated IT infrastructure. The results obtained show that the solution is capable of capturing similarity among activities in change plans, improving the accuracy of risk assessment for IT change planning.

There is a push in the enterprise towards facilitating processes from best practice frameworks (such as the IT Infrastructure Library (ITIL)) to make them more repeatable, efficient and cost-effective. Best practice processes provide descriptive, high level guidelines rather than prescriptive, precise process model definitions. They are meant to be followed by people and may be adapted and enacted differently in various realizations. Currently, ITIL processes are either supported by tools that hard code an interpretation of the process logic, or followed by people using productivity tools. This is inefficient because existing tools hardcode a rigid logic of the processes, and do not support collaborative and flexible realizations of processes. Moreover, there is a risk of information loss when people using rigid productivity tools, and are forced to collaborate outside of those tools. In this paper, we present a conversation-centered approach and a tool that enables dynamic and flexible definition and enactment of best practice processes in a collaborative and interactive manner. We address the issue of information loss by using the concept of a conversation as a container of information about the interactions among people in the context of a process. A conversation is backed with a semi-structured process model and process templates to support flexible and adaptive process realization. We showcase the approach using an illustrative use case in incident and problem management, based on best practice processes from ITIL.

There is great interest in building an accurate theoretical model of IT support organizations, for several purposes such as optimal workforce allocation and what-if scenario analysis. However, the complexity of real-life IT support organizations makes it extremely hard to model their organizational, structural and behavioral processes. While the adoption of stationary stochastic processes to model incident arrivals and of first-come-first-served GI/G/N queues to model support groups permits to reproduce with good enough fidelity the organization-wide behavior, this approach does not always accurately capture the internal dynamics of the organization. This paper presents an experimental analysis of transaction logs from a real-life IT support organization, provided to us by the Outsourcing Services Division of HP. The statistical analysis of transactional logs allows us to make some interesting considerations that can be used to build a more accurate model of the organization, as well as to gain useful experience in the modeling process.

Planning is a fundamental sub-process of the overarching Information Technology (IT) change management process, proposed by the Information Technology Infrastructure Library to help organizations to deploy and maintain IT services in an effective and efficient way. A major issue behind IT change planning and of special importance for the alignment of changes with business objectives/constraints - the adequate projection of which human resources to assign to change activities - has not been properly addressed in previous investigations. To fill this gap, in this paper we propose and analyze novel strategies for planning the assignment of human resources to change activities. These strategies explore different ways to prioritize humans to activities (i.e., from the most to the less efficient or proficient humans), and to rank/cluster the activities that should be analyzed first. The novel strategies have been experimentally evaluated through ChangeAdvisor, a prototypical implementation of a decision support system that helps IT administrators in the task of understanding the trade-offs between alternative change designs.

The context: the selection of a maximum value IT service portfolio is a major challenge to be faced by IT executives, during the Service Strategy phase of the IT service life cycle defined in ITIL. The problem: ITIL states that the value of an IT service should be estimated based on the improvement of customers' outcomes brought by the service. It is hard to apply this approach due to difficulties to access data related to customers' outcomes and to precisely isolate the influence of IT on the improvements of customers' performance. Our proposed solution: in this work, we propose the application of a formal business value model to quantify the business value of IT services, so as to pave the way for the definition of a maximum-value portfolio. The quantification method presented estimates the value of IT services through low-intrusion interactions between the IT Service provider and its customers. Validation: our approach was applied with success to the IT department of a multinational manufacturer. Results from the experiments enabled higher expected IT ROI through the use of a value-based portfolio definition process.

The number of changes that IT departments have to deal with is growing at a fast pace in response to changing business needs of enterprises. As changes are getting executed and deployed, knowledge is being created and stored. It is of paramount importance to the success of the business to re-use that knowledge for future changes. In fact, those who do not learn from past experiences are doomed to repeat the same mistakes as well as not bear the fruit of the ones that were successful. This paper addresses this concern by providing for every change being worked out the most similar past changes. Our solution combines data mining and optimization paradigms to model the problem of finding past similar changes by designing and learning similarity functions. Our approach enhances the efficiency and effectiveness of dealing with changes, by reducing the risk and shortening the time of introducing new changes.

Information Technology (IT) products and services provided by modern organizations are designed in projects that often involve large amount of resources (e.g., humans, hardware, and software). It is essential that organizations enforce rational practices for project management, in order to successfully conclude projects and avoid waste of substantial resources. In this context, Risk Management is fundamental to guarantee the accomplishment of project's objectives by dealing with adverse and favorable events. Although important, risk assessment in IT projects is usually performed by stakeholders in interviews and brainstorms which may be a very time/resource-consuming task. Therefore, in this paper, we introduce a solution to automate the risk assessment process, based on the history of previously conducted projects. Furthermore, comprehensive and interactive risk reports are proposed in order to ease the analysis of automatically generated reports. The results show that our solution is not only useful to speed the risk assessment process, but also to assist the decision making of project managers by organizing risk information according to the project structure.

Incident Management is the process through which IT support organizations manage to restore normal service operation after a service disruption. The complexity of real-life enterprise-class IT support organizations makes it extremely hard to understand the impact of organizational, structural and behavioral components on the performance of the currently adopted incident management strategy and, consequently, which actions could improve it. This paper presents SYMIAN, a decision support tool for the performance improvement of the incident management function in IT support organizations. SYMIAN simulates the effect of corrective measures before their actual implementation, enabling time, effort, and cost saving. To this end, SYMIAN models the IT support organization as an open queuing network, thereby enabling the evaluation of both the system-wide dynamics as well as the behavior of the individual organization components and their interactions. Experimental results show the SYMIAN effectiveness in the performance analysis and tuning of the incident management process for real-life IT support organizations.

Service disruptions can have a considerable impact on business operations of IT support organizations, thus calling for the implementation of efficient incident management and service restoration processes. The evaluation and improvement of incident management strategies currently in place, in order to minimize the business-impact of major service disruptions, is a very arduous task which goes beyond the optimization with respect to IT-level metrics. This paper presents HANNIBAL, a decision support tool for the business impact analysis and improvement of the incident management process. HANNIBAL evaluates possible strategies for an IT support organization to deal with major service disruptions. HANNIBAL then selects the strategy with the best alignment to the business objectives. Experimental results collected from the HANNIBAL application to a realistic case study show that business impact-driven optimization outperforms traditional performance-driven optimization.

The main goal of change management is to ensure that standardized methods and procedures are used for the efficient and prompt handling of changes in IT systems, in order to minimize change-related incidents and service-delivery disruption. To meet this goal, it is of paramount importance reusing the experience acquired from previous changes in the design of subsequent ones. Two distinct approaches may be usefully combined to this end. In a top-down approach, IT operators may manually design change templates based on the knowledge owned/acquired in the past. Considering a reverse, bottom-up perspective, these templates could be discovered from past execution traces gathered from IT provisioning tools. While the former has been satisfactorily explored in previous investigations, the latter - despite its undeniable potential to result in accurate templates in a reduced time scale - has not been subject of research, as far as the authors are aware of, by the service operations and management community. To fill in this gap, this paper proposes a solution, inspired on process mining techniques, to discover change templates from past changes. The solution is analyzed through a prototypical implementation of a change template miner subsystem called CHANGEMINER, and a set of experiments based on a real-life scenario.

In order to deal with failures in the deployment of IT changes and to always leave IT infrastructures into consistent states, we proposed in a previous work, a solution to automate the generation of rollback plans in IT change management systems. The solution was based on a mechanism that treats Requests for Change (RFC) (or parts of them) as a single atomic transaction. In this work, we extend our previous investigation and present more flexible and fine grained treatment of failures. The paper first presents extensions to our conceptual model in order (i) to give IT operators some flexibility in defining rollback actions, for example, by allowing the rollback plan to not only be a reversed change plan; and (ii) to execute different recovery activities depending on the cause and location of a problem. The paper then focuses on a refined manner to handle and treat failures in change deployments. We follow the ITIL version 3 best practises which suggest that, depending on the RFC context, the human operator can classify activities as reversible or irreversible. Such classification allows change management systems to automatically generate more accurate remediation plans. The proposal takes into account not only a precise way to define how rollback plans will be generated, but also an intuitive method enabling the operator to define compensation activities in order to complete the RFC successfully, even with the occurrence of failures. To prove the concept and technical feasibility, we have materialized our solution in the CHANGELEDGE prototype that, using elements of the Business Process Execution Language (BPEL), is able to generate correct remediation plans to handle and treat failures in IT change management systems.

The growing necessity of organizations in using technologies to support to their operations implies that managing IT resources became a mission-critical issue for the health of the primary companies' businesses. Thus, in order to minimize problems in the IT infrastructure, possibly affecting the daily business operations, risks intrinsic to the change process have to be analyzed and assessed. Risk Management is a widely discussed subject in several areas, although for IT Change Management it is quite a new discipline. The Information Technology Infrastructure Library (ITIL) introduces a set of best practices to conduct the management of IT infrastructures. According to ITIL, risks should be investigated, measured, and mitigated before any change is approved. Even with these guidelines, there is no default automatic method for risk assessment in IT Change Management. In this paper we introduce a risk analysis method based on the execution history of past changes. In addition, we propose a failure representation model to capture the feedback of the execution of changes over IT infrastructures.

This paper presents a model to support decision making for investments in IT services. As such it contributes to IT service portfolio management. Investment options are analyzed and ranked according to a utility index estimated from possible positive and negative business impact of IT services due to IT Service Level Agreement (SLA) compliance and violations. The Balanced Scorecard framework is used to infer the linkage between IT and business activities. The approach takes uncertainty into account by means of utility theory and interval arithmetic. Numerical illustrations encompassing tangible SLA-dependent criteria demonstrate how the approach may be of use.

Capturing and reusing the experience of operators in implementing IT changes is an important aspect of IT service management, as it may result in fewer incidents (upon change execution) and faster specification of change plans, to mention just a few potential advantages. Nevertheless, in practice, changes are usually described and documented in an ad hoc fashion, due to the lack of proper support to assist the design process. This hampers knowledge acquired when specifying, planning, and carrying out previous changes to be reused in subsequent requests. In order to address this issue, we propose the use of change templates as a mechanism to formalize, preserve, and reuse the experience accumulated within organizations in relation to IT changes. Our solution is analyzed through a prototypical implementation of a change management system and a case study based on a real-life scenario.

The current research on IT change management has been exploring several aspects of this new discipline, but it usually assumes that changes expressed in requests for change (RFC) documents will be successfully executed over the managed IT infrastructure. This assumption, however, is not realistic in actual IT systems because failures during the execution of changes do happen and cannot be ignored. In order to address this issue, we propose a solution where tightly-related change activities are grouped together forming atomic groups of activities. These groups are atomic in the sense that if one activity fails, all other already executed activities of the same group must rollback to move the system backwards to the previous state. The automation of change rollback is especially convenient because it relieves the IT human operator of manually undoing the activities of a change group that has failed. To prove concept and technical feasibility, we have materialized our solution in a prototype system that, using elements of the business process execution language (BPEL), is able to control how atomic groups of activities must be handled in IT change management systems.

In the Change Management process within IT Service Management, some activities need to evaluate the risk exposure associated with changes to be made to the infrastructure and services. The paper presents a method to evaluate risk exposure associated with a change. Further, we show how to use the risk exposure metric to automatically assign priorities to changes. The formal model developed for this purpose captures the business perspective by using financial metrics in the evaluation of risk. Thus the method is an example of Business-Driven IT Management. A case study, performed in conjunction with a large IT service provider, is reported and provides good results when compared to decisions made by human managers.

Change management is one of the most critical processes in IT management. Some of the reasons are the sheer number of changes and the difficulty of evaluating the impact of changes on the IT services being provided. Through carrying out a survey with IT managers and practitioners, we have found that, among the activities performed during change management, change scheduling (allocating changes to change windows) is the most problematic one. In this paper we solve the change scheduling problem by using a business-driven approach that evaluates the impact of a change schedule in terms of the financial loss imposed on the service provider. Toward this aim, we model the impact of SLA violations when the implementation of changes is done after their deadline. A change scheduling optimization problem is then formalized and its solution is applied to a typical scenario. The results show that optimizing the scheduling of changes can result in significant savings to an IT support organization.

In this paper we address the problem of ensuring business-IT alignment. We describe a method and a system for decision support in IT service management driven by alignment with the business objectives of the enterprise that the IT supports. Our technical proposition, called IT management by business objectives (MBO) is applicable to most of the domains of IT service management, such as incident management, change management, and others. The technology consists of some components that are reusable across domains, together with guidelines and patterns for building complementary components in order to develop domain-specific solutions

In this work, we have been principally concerned with the representation of contracts so that their normative state may be tracked in an automated fashion over their deployment lifetime. The normative state of a contract, at a particular time, is the aggregation of instances of normative relations that hold between contract parties at that time, plus the current values of contract variables. The effects of contract events on the normative state of a contract are specified using an XML formalization of the Event Calculus, called ecXML. We use an example mail service agreement from the domain of web services to ground the discussion of our work. We give a characterization of the agreement according to the normative concepts of obligation, power and permission, and show how the ecXML representation may be used to track the state of the agreement, according to a narrative of contract events. We also give a description of a state tracking architecture, and a contract deployment tool, both of which have been implemented in the course of our work.

Companies are increasingly seeking to manage their IT infrastructure in light of their business objectives. To address this need, we introduce management by contract (MbC) as a new paradigm for IT management. We propose a way of formalizing and analyzing contractual relationships in order to better inform IT-related decisions. In this paper, we present the information model that underpins MbC and describe our contract-based analysis algorithm.

No abstract available.

Service composition is the act of taking several component products or services, and bundling them together to meet the needs of a given customer. In the future, service composition will play an increasingly important role in e-commerce, and automation will be desirable to improve speed and efficiency of customer response. In this paper, we consider a service composition agent that both buys components and sells services through auctions. It buys component services by participating in many English auctions. It sells composite services by participating in Request-for-Quotes reverse auctions. Because it does not hold a long-term inventory of component services, it must take risks; it must make offers in reverse auctions prior to purchasing all the components needed, and must bid in English auctions prior to having a guaranteed customer for the composite good. We present algorithms that is able to manage this risk, by appropriately bidding/offering in many auctions and reverse auctions simultaneously. The algorithms will withdraw from one set of possible auctions and move to another set if this will produce a better-expected outcome, but will effectively manage the risk of accidentally winning outstanding bids/offers during the withdrawal process. We illustrate the behavior of these algorithms through a set of worked examples.

If an e-services approach to electronic commerce is to become widespread, standardisation of ontologies, message content and message protocols will be necessary. In this paper, we present a lifecycle of a business-to-business e-commerce interaction, and show how the Semantic Web can support a service description language that can be used throughout this lifecycle. By using DAML, we develop a service description language sufficiently expressive and flexible to be used not only in advertisements, but also in matchmaking queries, negotiation proposals and agreements. We also identify which operations must be carried out on this description language if the B2B lifecycle is to be fully supported. We do not propose specific standard protocols, but instead argue that our operators are able to support a wide variety of interaction protocols, and so will be fundamental irrespective of which protocols are finally adopted.