In this study, we explored an approach to automate the review process of software design documents by using LLM. We first analyzed the review methods of design documents and organized 11 review perspectives. Additionally, we analyzed the issues of utilizing LLMs for these 11 review perspectives and determined which perspectives can be reviewed by current general-purpose LLMs instead of humans. For the reviewable perspectives, we specifically developed new techniques to enable LLMs to comprehend complex design documents that include table data. For evaluation, we conducted experiments using GPT to assess the consistency of design items and descriptions across different design documents in the design process used in actual business operations. Our results confirmed that LLMs can be utilized to identify inconsistencies in software design documents during the review process.

Systems based on Deep Neural Networks (DNNs) are increasingly being used in industry. In the process of system operation, DNNs need to be updated in order to improve their performance. When updating DNNs, systems used in companies that require high reliability must have as few regressions as possible. Since the update of DNNs has a data-driven nature, it is difficult to suppress regressions as expected by developers. This paper identifies the requirements for DNN updating in industry and presents a case study using techniques to meet those requirements. In the case study, we worked on satisfying the requirement to update models trained on car images collected in Fujitsu assuming security applications without regression for a specific class. We were able to suppress regression by customizing the objective function based on NeuRecover, a DNN repair technique. Moreover, we discuss some of the challenges identified in the case study.

Additional training of a deep learning model can cause negative effects on the results, turning an initially positive sample into a negative one (degradation). Such degradation is possible in real-world use cases due to the diversity of sample characteristics. That is, a set of samples is a mixture of critical ones which should not be missed and less important ones. Therefore, we cannot understand the performance by accuracy alone. While existing research aims to prevent a model degradation, insights into the related methods are needed to grasp their benefits and limitations. In this talk, we will present implications derived from a comparison of methods for reducing degradation. Especially, we formulated use cases for industrial settings in terms of arrangements of a data set. The results imply that a practitioner should care about better method continuously considering dataset availability and life cycle of an AI system because of a trade-off between accuracy and preventing degradation.

Systematic techniques to improve quality of deep neural networks (DNNs) are critical given the increasing demand for practical applications including safety-critical ones. The key challenge comes from the little controllability in updating DNNs. Retraining to fix some behavior often has a destructive impact on other behavior, causing regressions, i.e., the updated DNN fails with inputs correctly handled by the original one. This problem is crucial when engineers are required to investigate failures in intensive assurance activities for safety or trust. Search-based repair techniques for DNNs have potentials to tackle this challenge by enabling localized updates only on “responsible parameters” inside the DNN. However, the potentials have not been explored to realize sufficient controllability to suppress regressions in DNN repair tasks. In this paper, we propose a novel DNN repair method that makes use of the training history for judging which DNN parameters should be changed or not to suppress regressions. We implemented the method into a tool called Neurecover and evaluated it with three datasets. Our method outperformed the existing method by achieving often less than a quarter, even a tenth in some cases, number of regressions. Our method is especially effective when the repair requirements are tight to fix specific failure types. In such cases, our method showed stably low rates (<2 %) of regressions, which were in many cases a tenth of regressions caused by retraining.

Significant effort is being put into developing industrial applications for artificial intelligence (AI), especially those using machine learning (ML) techniques. Despite the intensive support for building ML applications, there are still challenges when it comes to evaluating, assuring, and improving the quality or dependability. The difficulty stems from the unique nature of ML, namely, system behavior is derived from training data not from logical design by human engineers. This leads to black-box and intrinsically imperfect implementations that invalidate many principles and techniques in traditional software engineering. In light of this situation, the Japanese industry has jointly worked on a set of guidelines for the quality assurance of AI systems (in the Consortium of Quality Assurance for AI-based Products and Services) from the viewpoint of traditional quality-assurance engineers and test engineers. We report on the second version of these guidelines, which cover a list of quality evaluation aspects, catalogue of current state-of-the-art techniques, and domain-specific discussions in five representative domains. The guidelines provide significant insights for engineers in terms of methodologies and designs for tests driven by application-specific requirements.

Application Programming Interfaces (APIs) are common in software development to reuse other products. Although the documentation allows API consumers to learn about API usages, it can be unreliable. Here, we investigate the characteristics of inappropriate usage examples in web API documentation by extracting and comparing OpenAPI Specifications from usage example-response pairs. About 65.5% of the endpoints have some form of inappropriate usage examples. Furthermore, mismatches are classified into four categories: undocumented keys pattern, dynamic keys pattern, unreturned keys pattern, and type mismatched pattern. Our results suggest that the number of keys in the response is correlated with the number of mismatches. These findings should assist both API providers and consumers who deal with unreliable documentation in web APIs.

Test code is becoming more essential to the modern software development process. However, practitioners often pay inadequate attention to key aspects of test code quality, such as bug detectability, maintainability and speed. Existing tools also typically report a single test code quality measure, such as code coverage, rather than a diversified set of metrics. To measure and visualize quality of test code in a comprehensive fashion, we developed an integrated test code analysis tool called Phanta. In this show case, we posit that the enhancement of test code quality is key to modernizing software development, and show how Phanta's techniques measure the quality using mutation analysis, test code clone detection, and so on. Further, we present an industrial case study where Phanta was applied to analyze test code in a real Fujitsu project, and share lessons learned from the case study.

Mutation analysis is a method for evaluating the effectiveness of a test suite by seeding faults artificially and measuring the fraction of seeded faults detected by the test suite. The major limitation of mutation analysis is its lengthy execution time because it involves generating, compiling and running large numbers of mutated programs, called mutants. Our tool MuVM achieves a significant runtime improvement by performing higher order mutation analysis using four techniques, meta mutation, mutation on virtual machine, higher order split-stream execution, and online adaptation technique. In order to obtain the same behavior as mutating the source code directly, meta mutation preserves the mutation location information which may potentially be lost during bit code compilation and optimization. Mutation on a virtual machine reduces the compilation and testing cost by compiling a program once and invoking a process once. Higher order split-stream execution also reduces the testing cost by executing common parts of the mutants together and splitting the execution at a seeded fault. Online adaptation technique reduces the number of generated mutants by omitting infeasible mutants. Our comparative experiments indicate that our tool is significantly superior to an existing tool, an existing technique (mutation schema generation), and no-split-stream execution in higher order mutation.

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, we propose a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. The technique uses iterative, incremental refinement of test-drivers and symbolic execution, guided by a diagnostics engine. The diagnostics engine works off a novel precise and efficient byte-level dynamic dependence analysis built using Reduced Ordered Binary Decision Diagrams (ROBDDs). We present a tool FSX implementing this technique and evaluate it under two practical use-cases of incremental unit test generation, on five revisions of the open-source software iPerf, as well as on 3 large subjects, comprising more than 60 thousand lines of code, from in-house commercial network products. The evaluation shows that FSX can generate high-quality unit tests on large industrial software while minimizing the maintenance cost of the overall test-suite.

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, in previous work, we proposed a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. In this paper we describe a tool FSX, implementing this technique. We describe the architecture, user-interface, and salient features of FSX, and specific practical use-cases of its technology. We also report on a real, large-scale deployment of FSX, as a practical validation of the underlying research contribution and of automated test generation research in general.

After a legacy system is re-engineered, it is important to perform compatibility testing so as to identify the differences and reduce the introduced bugs. We can first apply symbolic execution to obtain an exhaustive set of test cases, then use them to check the compatibility of the old system and the new one. However there may be a lot of failed test cases which are a mix of erroneous and allowable incompatibilities. To locate the causes of failures detected during the testing, we apply multiple statistical bug localization techniques. We are able to localize 90% of the incompatibilities in 10% of the code for an industrial application with around 20k lines by Tarantula. And we identify the characteristics of failure causes which are difficult to be detected by statistical bug localization.

Symbolic execution as a test case generation technique has recently become an active research area. However, since symbolic execution generates a large number of test cases, it is impractical to run all the generated test cases in practice. In this paper, we present a test case selection method for a symbolic execution-based test case generation. This method has the following two characteristics. 1)Test cases which cover atomic conditions collected during symbolic execution, called path condition-based, are selected to keep the fault-detection capability. 2)This method does not depend on a particular symbolic execution engine since it is based on an analysis of path conditions in a general format. We implemented this method in our tool and evaluated this method with real systems. Our evaluation shows that the method produces a significant reduction in the size of the test suite while effectively preserving its fault-detection capability. We also demonstrate that the method is compatible with practically any symbolic execution engine, including the popular tools Java Path Finder and KLEE.

After a legacy system is re-engineered, it is important to perform compatibility testing so as to identify the difference and reduce the introduced bugs. We can first apply symbolic execution to obtain an exhaustive set of test cases, then use them to check the compatibility of the old system and the new old. However, the path explosion problem of symbolic execution makes it difficult to work on realistic non-trivial applications. We show in this paper how to enhance symbolic execution, e.g. with extra constraints, path cutting, variable grouping, and test case selection, to successfully test the compatibility of an SMTP library (used in embedded systems) with around 20K lines of code. Our experience indicates that these enhancements are essential to apply symbolic execution on realistic industrial applications.

In these days, there are a lot of massive event data appeared every day because networked devices such as smartphones and kind of sensors become very common. Every service provider intends to analyze the event data and discovers a new business rule for real time event processing service. We proposed a methodology which integrates such kind of data analysis and service development using DFD (Data Flow Diagram) which is independent from implementations, and we developed its prototype for evaluations. The methodology and the tool separate the concern of a developer and an analyst, moreover, we succeeded to confirm that the prototype tool reduced actual analysis and service development costs dramatically.