Systematic techniques to improve quality of deep neural networks (DNNs) are critical given the increasing demand for practical applications including safety-critical ones. The key challenge comes from the little controllability in updating DNNs. Retraining to fix some behavior often has a destructive impact on other behavior, causing regressions, i.e., the updated DNN fails with inputs correctly handled by the original one. This problem is crucial when engineers are required to investigate failures in intensive assurance activities for safety or trust. Search-based repair techniques for DNNs have potentials to tackle this challenge by enabling localized updates only on “responsible parameters” inside the DNN. However, the potentials have not been explored to realize sufficient controllability to suppress regressions in DNN repair tasks. In this paper, we propose a novel DNN repair method that makes use of the training history for judging which DNN parameters should be changed or not to suppress regressions. We implemented the method into a tool called Neurecover and evaluated it with three datasets. Our method outperformed the existing method by achieving often less than a quarter, even a tenth in some cases, number of regressions. Our method is especially effective when the repair requirements are tight to fix specific failure types. In such cases, our method showed stably low rates (<2 %) of regressions, which were in many cases a tenth of regressions caused by retraining.

Software development often involves systematic edits, similar but nonidentical changes to many code locations, that are error-prone and laborious for developers. Mining and learning such systematic edit patterns (SEPs) from past code changes enable us to detect and repair overlooked buggy code that requires systematic edits. A recent study presented a promising SEP mining technique that is based on program dependence graphs (PDGs), while traditional approaches leverage syntax-based representations. PDG-based SEPs are highly expressive and can capture more meaningful changes than syntax-based ones. The next challenge to tackle is to apply the same code changes as in PDG-based SEPs to other code locations; detection and repair of overlooked locations that require systematic edits. Existing program transformation techniques cannot well address this challenge because (1) they expect many structural code similarities that are not guaranteed in PDG-based SEPs or (2) they work on the basis of PDGs but are limited to specific domains (e.g., API migrations). We present in this paper a general-purpose program transformation algorithm for applying PDG-based SEPs. Our algorithm identifies a small transplantable structural subtree for each PDG node, thereby adapting code changes from PDG-based SEPs to other locations. We construct a program repair pipeline Sirius that incorporates the algorithm and automates the processes of mining SEPs, detecting overlooked code locations (bugs) that require systematic edits, and repairing them by applying SEPs. We evaluated the repair performance of Sirius with a corpus of open source software consisting of over 80 repositories. The results indicate that Sirius greatly outperformed the state-of-the-art technique for syntax-based SEPs. Sirius achieved a precision of 0.710, recall of 0.565, and F1-score of 0.630, while those of the state-of-the-art technique were 0.470, 0.141, and 0.216, respectively.

Programmers often use Q&A sites (e.g., Stack Overflow) to understand a root cause of program bugs. Runtime exceptions is one of such important class of bugs that is actively discussed on Stack Overflow. However, it may be difficult for beginner programmers to come up with appropriate keywords for search. Moreover, they need to switch their attentions between IDE and browser, and it is time-consuming. To overcome these difficulties, we proposed a method, "Q&A MAESTRO", to find suitable Q&A posts automatically for Java runtime exception by utilizing structure information of codes described in programming Q&A website. In this paper, we describe a usage scenario of IDE-plugin, the architecture and user interface of the implementation, and results of user studies. A video is available at https://youtu.be/4X24jJrMUVw. A demo software is available at https://github.com/FujitsuLaboratories/Q-A-MAESTRO.

Data augmentation techniques that increase the amount of training data by adding realistic transformations are used in machine learning to improve the level of accuracy. Recent studies have demonstrated that data augmentation techniques improve the robustness of image classification models with open datasets; however, it has yet to be investigated whether these techniques are effective for industrial datasets. In this study, we investigate the feasibility of data augmentation techniques for industrial use. We evaluate data augmentation techniques in image classification and object detection tasks using an industrial in-house graphical user interface dataset. As the results indicate, the genetic algorithm-based data augmentation technique outperforms two random-based methods in terms of the robustness of the image classification model. In addition, through this evaluation and interviews with the developers, we learned following two lessons: data augmentation techniques should (1) maintain the training speed to avoid slowing the development and (2) include extensibility for a variety of tasks.

Recent advances in automated program repair (APR) have widely caught the attention of industrial developers as a way of reducing debugging costs. While hundreds of studies have evaluated the effectiveness of APR on open-source software, industrial case studies on APR have been rarely reported; it is still unclear whether APR can work well for industrial software. This paper reports our experience applying a state-of-the-art APR technique, ELIXIR, to large industrial software consisting of 150+ Java projects and 13 years of development histories. It provides lessons learned and recommendations regarding obstacles to the industrial use of current APR: low recall (7.7%), lack of bug-exposing tests (90%), low success rate (10%), among others. We also report the preliminary results of our ongoing improvement of ELIXIR. With some simple enhancements, the success rate of repair has been greatly improved by up to 40%.

Virtualization technologies in a cloud computing system enable its users to create and delete virtual machines easily. This flexibility accelerates the update cycle of applications to keep up with user demand. When updating applications, the application management process should also be modified. If necessary changes in the management process are overlooked, this lack of changes can cause inconsistencies between an application and its management, resulting in improper system status. To identify required changes in management process, we propose a method to identify them based on an assumption that similar application updates often require similar operation changes. Evaluation results show that our method can recommend the necessary operation changes with high accuracy and can reduce the possibility of operators overlooking them about 63% of the time if operation changes are overlooked.

Generally, the automation operations for cloud management achieved by replacing manual operations with operations using automation tools. The developers of automation scripts often refer to the existing automation scripts so that they can develop new automation scripts by just modifying smaller parts in the existing automation scripts. In order to facilitate the development of automation scripts, we propose a method of appropriately identifying the existing automation scripts to refer to in developing automation scripts.

Recent studies identify misconfiguration as the most frequent cause for failures in information systems including cloud computing infrastructure. Therefore, reducing operator error is key to improving the reliability and availability of cloud services. In order to reduce the risk of improper configurations, we propose the following two techniques; (1) automated synthesis of configuration change procedure to avoid improper changes and (2) identification of vulnerabilities (e.g. single point of failures) in system configuration to increase service resilience in the presence of undesirable events such as components failures and improper operations. We devised frameworks that realize these techniques using formal methods and evaluate their effectiveness through case studies. Based on the results of this evaluation, we discuss the benefits and limitations in applying formal methods for cloud system managements.

To manage cloud computing infrastructures consisting of many servers having a massive number of configuration parameters is quite burdensome for administrators of infrastructures. While some policy-based management approaches have been proposed to maintain the system configuration, it is quite difficult for administrators to define proper configuration policies for parameter settings in large-scale cloud computing infrastructure. To solve this problem, we developed a method of extracting parameter configuration policies from the configuration information of the existing infrastructure using UML/OCL verification. In this method, first we identify the scopes of management from the hierarchical topology of the cloud infrastructure. Next, we execute verifications of two types of OCL constraints (regarding parameter configuration patterns) for the configuration of the infrastructure, while changing the range of the scopes we focus on. By determining whether or not these constraints can be satisfied with some scopes, we extract policies that represent patterns satisfied between parameter settings of servers deployed in a certain range of the scope. Then, we demonstrate that we can derive configuration policies for all of parameters of servers in an actual cloud service infrastructure through a case study.

Large-scale cloud data centers have a great number of configuration parameters, and as such it is difficult for administrators to configure these parameters correctly. Proposals have been made for approaches to automatic parameter configuration focusing mainly on the identification of common configurations in existing infrastructures as "common design patterns" and applying these patterns to the development of new cloud infrastructure. However, these methods still require manual configuration for the areas which do not have common design patterns, and manual configuration must be reduced as far as possible because misconfiguration is one of the most dominant causes of service failures. Taking this background into account, here we propose an automatic configuration method which reduces manual configuration by identifying "incremental design patterns" which are linear relationships between generations of existing infrastructures (the time the infrastructures were constructed) and the values assigned to the parameters. Furthermore, we construct a parameter configuration procedure by ordering the application of design patterns. For the design patterns that are interdependent, we resolve them by recursively applying decision tree analysis. By using these design patterns and application procedure, we can reduce the manual configuration that is necessary. We evaluated our method in actual cloud infrastructures and confirmed that the proposed approach could configure 91.3% of parameters in new infrastructure automatically.

Live migration technologies can contribute to efficient resource management in a cloud datacenter; however, they will inevitably entail downtime for the virtual machine involved. Even if the downtime is relatively short, its effect can be serious for applications sensitive to response time degradations. Therefore, cloud datacenter providers should control live migration operations to minimize the impact on the performance of applications running on the cloud infrastructure. With this understanding, we studied the impact of live migration on the performance of 2-tier web applications in an experimental setup using XenServer and RUBBoS benchmark. We revealed that the behavior of the transmission control protocol (TCP) can be the primary factor responsible for response time degradation during live migration. On the basis of the experimental results, we constructed functions to estimate the performance impact of live migration on the applications. We also examined a case study to demonstrate how cloud computing datacenters can determine the best live migration strategy to minimize application performance degradation.

Since many components comprising large scale cloud datacenters have a great number of configuration parameters (e.g. hostnames, languages, and time zones), it is difficult to keep consistencies in the configuration parameters. In such cases, misconfigured parameters can cause service failures. For this reason, we propose a misconfiguration detection method for large-scale cloud datacenters, which can automatically determine possible misconfigurations by identifying the relations existing among majority of the parameters using statistical decision tree analysis. We have also developed a pattern modification method to improve the accuracy of the decision tree approach. We evaluated the misconfiguration detection performance of the proposed method by using both artificial data and actual data. The results show that we can achieve higher accuracy (78.6% in the actual data) in misconfiguration detection by using the pattern modification.

Once failures occur in a cloud datacenter accommodating a large number of virtual resources, they tend to spread rapidly and widely, impacting on many cloud users (tenant owners). One of the best ways to prevent a failure from spreading in the system is identifying signs of the failure before its occurrence and deal with it proactively before it causes serious problems. Although several approaches have been proposed to predict failures by analyzing past system message logs and identifying the relationship between the messages and the failures, it is still difficult to automatically predict the failure for several reasons such as various types of log message formats or time gaps between message pattern learning and application of the identified patterns in real systems. Based on this understanding, we propose a new failure prediction method in this paper which learns message patterns as the signs of failure automatically by classifying messages by their similarity without depending on their format and re-Iearning of message patterns in frequently-changed configurations. We implemented our failure prediction method and evaluated it by using system log data recorded in an actual cloud datacenter. The experimental result shows that our approach predicted failures with 80% precision and covered 90% of failure occurrences.

Server virtualization technologies and their live migration function contribute to the utilization of the computing resources in cloud datacenters. However, many management operations for virtual machines (VMs) including live migrations can be evoked by many cloud users at anytime. The outburst of executions of live migration operations can deteriorate migration performance and make it difficult to provide required resources to users in a timely manner. Therefore, understanding the behaviors and the performance of simultaneous live migrations is very important to provide efficient and reliable cloud computing services. In this paper, we construct a performance model of concurrent live migrations in virtualized datacenters. We first collect performance data from an experimental virtualized system in which we execute simultaneous live migrations. Based on the data, we next construct a performance model representing the performance characteristics of live migration using PRISM, a probabilistic model checker. We then demonstrate that we can easily verify the properties described in PRISM language regarding live migration performance using this model.

Currently, cloud data centers make use of server virtualization techniques that consolidate computational resources to provide various services and improve resource utilization. However, the pattern of resource usage in some management operations for virtualized systems, such as live migration and snapshot recording, is different from that in the case of a traditional data center in which the virtualization technology is not used. Therefore, understanding the system behavior during the execution of a large number of management operations is very important for the reliable management of a virtualized cloud data center. With this understanding, we studied the characteristics of management operation performance by executing many operations simultaneously in an experimental virtual server system. The experimental results revealed some notable characteristics, including interference between operations with virtual machines (VMs) hosted on different physical servers and the asymmetric nature of live migration. On the basis of these findings, we state that degradation of operation performance can be mitigated by orchestrating operations under a proper control policy. We confirm the validity of these suggestions by carrying out a case study.

Managing the configurations of complex systems consisting of various components requires combined efforts by multiple domain experts. These experts have extensive knowledge about different components in the system they need to manage, but little understanding of the issues outside their individual areas of expertise. As a result, the configuration constraints, changes, and procedures specified by those involved in the management of a complex system are often interrelated with one another without being noticed, and their integration into a coherent procedure for configuration represents a major challenge. The method of synthesizing the configuration procedure introduced in this paper addresses this challenge using a combination of formal specification and model finding techniques. We express the knowledge on system management with this method, which is provided by domain experts as first-order logic formulas in the Alloy specification language, and combine it with system-configuration information and the resulting specification. We then employ the Alloy Analyzer tool to find a system model that satisfies all the formulas in this specification. The model obtained corresponds to a procedure for system configurations that satisfies all expert-specified constraints. Finally, we evaluate our method through a case study on a procedure to consolidate virtual machines.

Network services based on distributed components, which provide advanced network services with lower cost by assigning and reusing useful components running on remote nodes, are attracting increasing attention. In this service, when a component fails, a component that has the same function running on another node can be substituted for it. On the other hand, there are three kinds of failures in network services; a software failure at a component, a hardware failure at a node, and a failure of the physical network. We propose a new method to identify a failure in network services, which determines not only a failure at the application level but also in the network layer by collecting a small number of messages via multiple overlay networks.

It is a common task for many system managers to analyze performance problems to keep the system response time small, but it is difficult to identify the characteristics of the situations in which performance problems occur. To solve this problem, we developed an analysis method that uses a decision tree approach for Web system performance problems. In order to output simpler and more understandable results, we developed an original evaluation function for tree construction rather than using an existing one. In addition, to construct an appropriate decision tree from a huge number of parameters, we developed an algorithm which repeats decision tree construction and evaluation with a changing set of construction parameters and chooses the best tree. We implemented and evaluated our algorithm in an actual Web system and we found that our approach is much better than extant decision tree algorithms in terms of accuracy, simplicity, and computational time.

To provide a consistent quality of service in IP networks, it is necessary to know the current performance and available bandwidth of the network adequately. There are some ways to obtain the current available bandwidth such as management information base (MIB) analysis and RMON monitoring. The measurement methods using probing packets are studied, but they require a large number of packets to be transmitted in order to improve the measurement accuracy. They create heavy network traffic, so it is difficult to use such methods for actual networks in use. To solve the problem, we propose a new method called NEPRI. This method detects the macroscopic behaviour of the probing packets queued at the bottleneck and estimates the available bandwidth of the path. We show our method consumes less network resources and is sufficiently accurate.