Memory failures frequently lead to server break-downs in large-scale datacenters, with uncorrectable errors (UEs) serving as primary indicators of defects in Dual Inline Memory Modules (DIMMs). Existing approaches mainly focus on predicting UEs using correctable errors (CEs), but often neglect the relationships of these errors across various CPU architectures, especially in the context of Error Correction Code (ECC). In this paper, we explore the correlation between CEs and UEs across different CPU Architectures, such as x86 and ARM. Our analysis reveals distinctive failure patterns in memory specific to each processor platform. Leveraging Machine Learning (ML) techniques on production datasets, we demonstrate that our approach substantially enhances prediction performance up to 15% in F1-score compared to the existing alaorithm.

The increasing demand for cloud computing drives the expansion in scale of datacenters and their internal optical network, in a strive for increasing bandwidth, high reliability, and lower latency. Optical transceivers are essential elements of optical networks, whose reliability has not been well-studied compared to other hardware components. In this paper, we leverage high quantities of monitoring data from optical transceivers and OS-level metrics to provide statistical insights about the occurrence of optical transceiver failures. We estimate transceiver failure rates and normal operating ranges for monitored attributes, correlate early-observable patterns to known failure symptoms, and finally develop failure prediction models based on our analyses. Our results enable network administrators to deploy early-warning systems and enact predictive maintenance strategies, such as replacement or traffic re-routing, reducing the number of incidents and their associated costs.

In large-scale datacenters, memory failure is one of the leading causes of server crashes, and uncorrectable error (UCE) is the major fault type indicating defects of memory modules. Existing approaches tend to predict UCEs using Correctable Errors (CE). However, bit-level CE information has not been completely discussed in previous works and CEs with error bit patterns are strongly correlated with UCE occurrences. In this paper, we present a novel Hierarchical Intelligent Memory Failure Prediction (HiMFP) framework which can predict UCEs on multiple levels of the memory system and associate with memory recovery techniques. Particularly, we leverage CE addresses on multiple levels of memory, especially bit-level, and construct machine learning models based on spatial and temporal CE information. Results of algorithm evaluation using real-world datasets indicate that HiMFP significantly enhances the prediction performance compared with the baseline algorithm. Overall, Virtual Machines (VM) interruptions caused by UCEs can be reduced by around 45% using HiMFP.

Excessive network packet loss is one of the strongest symptoms of the presence of some hardware or software-related infrastructure anomaly. As such, multiple techniques to rapidly and efficiently measure packet loss in physical network equipment have been developed throughout the years. Yet, most of the proposed techniques fall short in the case of cloud scenarios which combine the presence of physical and virtual network devices. In this paper, we tackle the problem of providing lightweight and fine-grained packet loss monitoring at virtual switches. We achieve our goal by combining packet coloring, with efficient packet loss signal extraction and aggregation entirely within the virtual switch level. To understand its feasibility in a production environment, the proposed system has been implemented and evaluated in a synthetic scenario and for real-world use cases. Our implementation on top of OvS-DPDK shows that the proposed system achieves 95% accurate packet loss measurement while introducing negligible switching throughput degradation.

In large-scale datacenters, memory failure is a common cause of server crashes, with uncorrectable errors (UEs) being a major indicator of Dual Inline Memory Module (DIMM) defects. Existing approaches primarily focus on predicting UEs using correctable errors (CEs), without fully considering the information provided by error bits. However, error bit patterns have a strong correlation with the occurrence of uncorrectable errors (UEs). In this paper, we present a comprehensive study on the correlation between CEs and UEs, specifically emphasizing the importance of spatio-temporal error bit information. Our analysis reveals a strong correlation between spatio-temporal error bits and UE occurrence. Through evaluations using real-world datasets, we demonstrate that our approach significantly improves prediction performance by 15% in F1-score compared to the state-of-the-art algorithms. Overall, our approach effectively reduces the number of virtual machine interruptions caused by UEs by approximately 59%.

Accurate, timely Root Cause Analysis (RCA) is essential to successful IT operations as a primary step to incident remediation. RCA automation using data mining techniques in large heterogeneous systems is, however, a challenging task, because it requires correlating multimodal information across various data sources. An increasing number of services are migrating to structured logging to enable automated monitoring and debugging of complex large-scale systems. In this paper, we leverage structured logs and association rule mining (ARM) to automate RCA. We propose the LogRule algorithm, which automatically analyzes structured logs to generate a list of explanations for an event of interest. It achieves 0.921 F1-score for the diagnosis task, while computing results 37x faster compared to the state-of-the-art solution based on FP-growth, making it a time-efficient, accurate, and interpretable ARM-based RCA algorithm. Evaluation results show that LogRule enables RCA in complex multidimensional datasets, where the execution time of the current state-of-the-art algorithm is prohibitively large.

Dynamic random access memory failures are a threat to the reliability of data centres as they lead to data loss and system crashes. Timely predictions of memory failures allow for taking preventive measures such as server migration and memory replacement. Thereby, memory failure prediction prevents failures from externalizing, and it is a vital task to improve system reliability. In this paper, we revisited the problem of memory failure prediction. We analyzed the correctable errors (CEs) from hardware logs as indicators for a degraded memory state. As memories do not always work with full occupancy, access to faulty memory parts is time distributed. Following this intuition, we observed that important properties for memory failure prediction are distributed through long time intervals. In contrast, related studies, to fit practical constraints, frequently only analyze the CEs from the last fixed-size time interval while ignoring the predating information. Motivated by the observed discrepancy, we study the impact of including the overall (long-range) CE evolution and propose novel features that are calculated incrementally to preserve long-range properties. By coupling the extracted features with machine learning methods, we learn a predictive model to anticipate upcoming failures three hours in advance while improving the average relative precision and recall for 21% and 19% accordingly. We evaluated our methodology on real-world memory failures from the server fleet of a large cloud provider, justifying its validity and practicality.

The reliability of cloud platforms is of significant relevance because society increasingly relies on complex software systems running on the cloud. To improve it, cloud providers are automating various maintenance tasks, with failure identification frequently being considered. The precondition for automation is the availability of observability tools, with system logs commonly being used. The focus of this paper is log-based failure identification. This problem is challenging because of the instability of the log data and the incompleteness of the explicit logging failure coverage within the code. To address the two challenges, we present CLog as a method for failure identification. The key idea presented herein based is on our observation that by representing the log data as sequences of subprocesses instead of sequences of log events, the effect of the unstable log data is reduced. CLog introduces a novel subprocess extraction method that uses context-aware neural network and clustering methods to extract meaningful subprocesses. The direct modeling of log event contexts allows the identification of failures with respect to the abrupt context changes, addressing the challenge of insufficient logging failure coverage. Our experimental results demonstrate that the learned subprocesses representations reduce the instability in the input, allowing CLog to outperform the baselines on the failure identification subproblems - 1) failure detection by 9–24 % on F1 score and 2) failure type identification by 7% on the macro averaged F1 score. Further analysis shows the existent negative correlation between the instability in the input event sequences and the detection performance in a model-agnostic manner.

In the current IT world, developers write code while system operators run the code mostly as a black box. The connection between both worlds is typically established with log messages: the developer provides hints to the (unknown) operator, where the cause of an occurred issue is, and vice versa, the operator can report bugs during operation. To fulfil this purpose, developers write log instructions that are structured text commonly composed of a log level (e.g., "info", "error"), static text ("IP {} cannot be reached"), and dynamic variables (e.g. IP {}). However, opposed to well-adopted coding practices, there are no widely adopted guidelines on how to write log instructions with good quality properties. For example, a developer may assign a high log level (e.g., "error") for a trivial event that can confuse the operator and increase maintenance costs. Or the static text can be insufficient to hint at a specific issue. In this paper, we address the problem of log quality assessment and provide the first step towards its automation. We start with an in-depth analysis of quality log instruction properties in nine software systems and identify two quality properties: 1) correct log level assignment assessing the correctness of the log level, and 2) sufficient linguistic structure assessing the minimal richness of the static text necessary for verbose event description. Based on these findings, we developed a data-driven approach that adapts deep learning methods for each of the two properties. An extensive evaluation on large-scale open-source systems shows that our approach correctly assesses log level assignments with an accuracy of 0.88, and the sufficient linguistic structure with an F1 score of 0.99, outperforming the baselines. Our study highlights the potential of the data-driven methods in assessing log instructions quality.

Modern society is increasingly moving toward complex and distributed computing systems. The increase in scale and complexity of these systems challenges O&M teams that perform daily monitoring and repair operations, in contrast with the increasing demand for reliability and scalability of modern applications. For this reason, the study of automated and intelligent monitoring systems has recently sparked much interest across applied IT industry and academia. Artificial Intelligence for IT Operations (AIOps) has been proposed to tackle modern IT administration challenges thanks to Machine Learning, AI, and Big Data. However, AIOps as a research topic is still largely unstructured and unexplored, due to missing conventions in categorizing contributions for their data requirements, target goals, and components. In this work, we focus on AIOps for Failure Management (FM), characterizing and describing 5 different categories and 14 subcategories of contributions, based on their time intervention window and the target problem being solved. We review 100 FM solutions, focusing on applicability requirements and the quantitative results achieved, to facilitate an effective application of AIOps solutions. Finally, we discuss current development problems in the areas covered by AIOps and delineate possible future trends for AI-based failure management.

The detection of anomalies is an essential data mining task for achieving security and reliability in computer systems. Logs are a common and major data source for anomaly detection methods in almost every computer system. Recent studies have focused predominantly on one-class deep learning methods on manually specified log representations. The main limitation is that these models are not able to learn log representations describing the semantic differences between normal and anomaly logs, leading to a poor generalization on unseen logs. We propose Logsy, a classification-based method to learn log representations that allow to distinguish between normal system log data and anomaly samples from auxiliary log datasets, easily accessible via the internet. The idea behind such an approach to anomaly detection is that the auxiliary dataset is sufficiently informative to enhance the representation of the normal data, yet diverse to regularize against overfitting and improve generalization. We perform several experiments on publicly available datasets to evaluate the performance and properties, where we show improvement of 0.25 in F1 compared to previous methods.

The concept of Artificial Intelligence for IT Operations (AIOps) combines big data and machine learning methods to replace a broad range of IT operations including availability and performance monitoring of services. Such platforms typically use separate models for each modality of monitoring data (e.g., textual properties and real-valued response time in logs and traces) to detect faults and upcoming anomalies in cloud services, which do not capture the existing correlation between the modalities. This paper extends the range of utilized data types for creation of a single model to improve the anomaly detection. We use a bimodal distributed tracing data from large cloud infrastructures in order to detect an anomaly in the execution of system components. We propose an anomaly detection method, which utilizes a single modality of the data with information about the trace structure. In the next step, we extend the single-modality neural architecture to a multimodal neural network with long short-term memory (LSTM) to enable the learning from the sequential nature of both modalities in the tracing data. Furthermore, we demonstrate an approach to detect dependent and concurrent events using the ability of the model to reconstruct the execution path. The implemented prototype is experimentally evaluated with data from a large-scale production cloud. The results demonstrate that the novel approaches outperform other deep-learning methods based on traditional architectures.

Artificial Intelligence for IT Operations (AIOps) combines big data and machine learning to replace a broad range of IT Operations tasks including availability, performance, and monitoring of services. By exploiting log, tracing, metric, and network data, AIOps enable detection of faults and issues of services. The focus of this work is on detecting anomalies based on distributed tracing records that contain detailed information for the availability and the response time of the services. In large-scale distributed systems, where a service is deployed on heterogeneous hardware and has multiple scenarios of normal operation, it becomes challenging to detect such anomalous cases. We address the problem by proposing unsupervised, response time anomaly detection based on deep learning data modeling techniques; unsupervised dynamic error threshold approach; tolerance module for false positive reduction; and descriptive classification of the anomalies. The evaluation shows that the approach achieves high accuracy and solid performance in both, experimental testbed and large-scale production cloud.

Context. Improving the efficiency and effectiveness of software development projects implies understanding their actual process. Given the same requirements specification, different software development teams may follow different strategies and that may lead to inappropriate use of tools or non-optimized allocation of effort on spurious activities, non-aligned with the desired goals. However, due to its intangibility, the actual process followed by each developer or team is often a black box. Objective. The overall goal of this study is to improve the knowledge on how to measure efficiency in development teams where a great deal of variability may exist due to the human-factor. The main focus is on the discovery of the underlying processes and compare them in terms of efficiency and effectiveness. By doing so, we expect to reveal potentially hidden costs and risks, so that corrective actions may take place on a timely manner during the software project life cycle. Method. Several independent teams of Java programmers, using the Eclipse IDE, were assigned the same software quality task, related to code smells detection for identifying refactoring opportunities and the quality of the outcomes were assessed by independent experts. The events corresponding to the activity of each team upon the IDE, while performing the given task, were captured. Then, we used process mining techniques to discover development process models, evaluate their quality and compare variants against a reference model used as "best practice". Results. Teams whose process model was less complex, had the best outcomes and vice-versa. Comparing less complex process variants with the ""best practice"" process, showed that they were also the ones with less differences in the control-flow perspective, based on activities frequencies. We have also determined which teams were most efficient through process analysis. Conclusions. We confirmed that, even for a well-defined software development task, there may be a great deal of process variability due to the human factor. We were able to identify when developers were more or less focused in the essential tasks they were required to perform. Less focused teams had the more complex process models, due to the spurious / non-essential actions that were carried out. In other words, they were less efficient. Experts' opinion confirmed that those teams also were less effective in their expected delivery. We therefore concluded that a self-awareness of the performed process rendered by our approach, may be used to identify corrective actions that will improve process efficiency (less wasted effort) and may yield to better deliverables, i.e. improved process effectiveness.

Measuring the capacity and modeling the response to load of a real distributed system and its components requires painstaking instrumentation. Even though it greatly improves observability, instrumentation may not be desirable, due to cost, or possible due to legacy constraints. To model how a component responds to load and estimate its maximum capacity, and in turn act in time to preserve quality of service, we need a way to measure component occupation. Hence, recovering the occupation of internal non-instrumented components is extremely useful for system operators, as they need to ensure responsiveness of each one of these components and ways to plan resource provisioning. Unfortunately, complex systems will often exhibit non-linear responses that resist any simple closed-form decomposition. To achieve this decomposition in small subsets of non-instrumented components, we propose training a neural network that computes their respective occupations. We consider a subsystem comprised of two simple sequential components and resort to simulation, to evaluate the neural network against an optimal baseline solution. Results show that our approach can indeed infer the occupation of the layers with high accuracy, thus showing that the sampled distribution preserves enough information about the components. Hence, neural networks can improve the observability of online distributed systems in parts that lack instrumentation.

As the provision of services and the use of personal data expands, the need for services to explicitly detail what personal data a service handles and in which manner becomes paramount in order to achieve a fully transparent, ethical and personalized user experience. Services usually require access to sensitive information and may distribute this information to third parties. Service consumers need to be informed about the ways their data are used and about the actors involved in this process. Universal service descriptions that can be used to cover any business service are required to provide interoperability. In this paper, we describe our work on the privacy module for the Linked Unified Service Description Language (USDL). We expand the language by introducing a new module that allows the specification of privacy properties for business services. We have considered recent advances in data protection for its creation and provide a method, accompanied by a software tool, to examine the validity of privacy policy descriptions with Linked USDL Privacy module.

In pursuit of faster development cycles, companies have favored small decoupled services over monoliths. Following this trend, distributed systems made of microservices have grown in scale and complexity, giving rise to a new set of operational problems. Even though this paradigm simplifies development, deployment, management of individual services, it hinders system observability. In particular, performance monitoring and analysis becomes more challenging, especially for critical production systems that have grown organically, operate continuously, and cannot afford the availability cost of online benchmarking. Additionally, these systems are often very large and expensive, thus being bad candidates for full-scale development replicas. Creating models of services and systems for characterization and formal analysis can alleviate the aforementioned issues. Since performance, namely response time, is the main interest of this work, we focused on bottleneck detection and optimal resource scheduling. We propose a method for modeling production services as queuing systems from request traces. Additionally, we provide analytical tools for response time characterization and optimal resource allocation. Our results show that a simple queuing system with a single queue and multiple homogeneous servers has a small parameter space that can be estimated in production. The resulting model can be used to accurately predict response time distribution and the necessary number of instances to maintain a desired service level, under a given load.

Despite the advantages of microservice and function-oriented architectures, there is an increase in complexity to monitor such highly dynamic systems. In this paper, we analyze two distinct methods to tackle the monitoring problem in a system with reduced instrumentation. Our goal is to understand the feasibility of such approach with one specific driver: simplicity. We aim to determine the extent to which it is possible to characterize the state of two generic tandem processes, using as little information as possible. To answer this question, we resorted to a simulation approach. Using a queue system, we simulated two services, that we could manipulate with distinct operation sets for each module. We used the total response time seen upstream of the system. Having this setup and metric, we applied two distinct methods to analyze the results. First, we used supervised machine learning algorithms to identify where the bottleneck is happening. Secondly, we used an exponential decomposition to identify the occupation in the two components in a more black-box fashion. Results show that both methodologies have their advantages and limitations. The separation of the signal more accurately identifies occupation in low occupied resources, but when a service is totally dominating the overall time, it lacks precision. The machine learning has a more stable error, but needs the training set. This study suggest that a black-box occupation approach with both techniques is possible and very useful.

Nowadays, service trading over the Web is gaining momentum. In this highly dynamic scenario, both providers and consumers need to formalize their contractual and legal relationship, creating service level agreements. Although there exist some proposals that provide models to describe that relationship, they usually only cover technical aspects, not providing explicit semantics to the agreement terms. Furthermore, these models cannot be effectively shared on the Web, since they do not actually follow Web principles. These drawbacks hamper take-up and automatic analysis. In this article, we introduce Linked USDL Agreement, a semantic model to specify, manage and share service level agreement descriptions on the Web. This model is part of the Linked USDL family of ontologies that can describe not only technical but also business related aspects of services, incorporating Web principles. We validate our proposal by describing agreements in computational and non-computational scenarios, namely cloud computing and business process outsourcing services. Moreover, we evaluate the actual coverage and expressiveness of Linked USDL Agreement comparing it with existing models. In order to foster its adoption and effectively manage the service level agreement lifecycle, we present an implemented tool that supports creation, automatic analysis, and publication on the Web of agreement descriptions.

MOOC platforms are Web-based learning environments which allow a global participation on a large scale and with free access. The paradigm presents itself as a new teaching trend, changing the way education can be offered and funded worldwide. Many institutions are now investing in this teaching mode. However, since it is a web-based tool, the connection performance can impact both the way learning is experienced by the student as well as the operation of the platform. The “Quality of Experience” (QoE) concept has been widely used to refer to how users describe a service they have used while the “Quality of Service” (QoS) concept deals with the technical performance parameters that are associated with the connection quality. This paper starts the process of developing a systematic mapping around MOOC platforms, and QoS and QoE concepts, aiming to provide an overview of the current state of research on these issues.

As the use of services available on the Web is becoming mainstream, contracts and legal aspects of the relationship between providers and consumers need to be formalized. However, current proposals to model service level agreements are mostly focused on technical aspects, do not explicitly provide semantics to agreement terms, and do not follow Web principles. These limitations prevent take-up, automatic processing, and effective sharing of agreements. Linked USDL Agreement is a Linked Data based semantic model to describe and share service agreements that extends Linked USDL, which offers a family of languages to describe various technical and business aspects of services. We followed a use case driven approach, evaluating the applicability of our proposal in a cloud computing scenario, and comparing its expressiveness with existing models. Finally, we show a concrete tool that helps to model and check the validity of agreements.

The most industrialized countries have entered an era where their prosperity is largely created through a service economy. There is a transition from a manufacturing based economy to a service based economy. In the Internet of Services (IoS) vision services are seen as tradable goods. Business services are a major asset in this context. Since they are inherently different from e.g. web services, there is a need for a specialized monitoring approach. In this paper we describe the differences between e-services and business services, present an approach to describe business services, and discuss challenges regarding their monitoring requirements.

Systems and infrastructures are currently being developed to support Web services and Web processes. One prominent solution to manage and coordinate Web services is the use of workflow technology. For more than two decades now, workflow management systems architectures, language specifications, and workflow analysis techniques have been extensively studied. While these areas of research have made the development of sophisticated workflow systems possible, one important research area that has been overlooked is the lifecycle of process application development. As a result, current process management systems support the analysis, enactment, and ad hoc design of workflows, but they lack the tools and methods to assist process design. The purpose of our study is to present a framework to assist and guide process analysts and designers in their task. The Poseidon framework includes a participative and an analytical design method. The participative phase uses a clean sheet approach and starts by constructing a business case table which captures all the business cases represented by a process. Afterward, an analytical design phase is followed where scheduling functions are derived from the business case table and are used to build the structure of a process.

Process measurement is the task of empirically and objectively assigning numbers to the attributes of processes in such a way as to describe them. We define process complexity as the degree to which a process is difficult to analyze, understand or explain. One way to analyze a process' complexity is to use a process control-flow complexity measure. This measure analyzes the control-flow of processes and can be applied to both Web processes and workflows. In this paper, we discuss how to evaluate the control-flow complexity measure to ensure that it can be qualify as a good and comprehensive one.

This paper deals with the evolution of inter-enterprise and Web scale process to support e-commerce and e-services. It taps into the promises of two of the hottest R&D and technology areas: Web services and the semantic Web. It presents how applying semantics to each of the steps in the semantic Web process lifecycle can help address critical issues in reuse, integration and scalability.