ChatGPT presents emerging opportunities in software development, yet its capabilities for understanding code remain largely understudied. This study aims to focus on the logical aspect of code comprehension of ChatGPT by examining its performance in detecting and fixing bugs. Our preliminary results suggest that ChatGPT seems to correct code in a different way than human logical steps.

Exploratory testing is an important activity for improving software quality. Despite this, few studies have been conducted to support exploratory testing. One reason for this is the lack of available data on exploratory testing activities. We propose LatteArt, a tool to support developers in recording, visualizing and managing exploratory testing activities. LatteArt has a rich user interface and a variety of features to support exploratory testing, and is publicly available. We conducted a questionnaire with three developers to assess whether LatteArt addresses the challenges of exploratory testing. In our questionnaire, two or more developers indicated that LatteArt helps to address 14 of the 17 exploratory testing weaknesses identified in the existing literature. The test activity data recorded by LatteArt is easily accessible via REST APIs, allowing researchers to use the data and facilitate exploratory testing research. Our tool and demo are available at: https://github.com/latteart-org/latteart

Automated program generation (APG) is a concept of automatically making a computer program. Toward this goal, transferring automated program repair (APR) to APG can be considered. APR modifies the buggy input source code to pass all test cases. APG regards empty source code as initially failing all test cases, i.e., containing multiple bugs. Search-based APR repeatedly generates program variants and evaluates them. Many traditional APR systems evaluate the fitness of variants based on the number of passing test cases. However, when source code contains multiple bugs, this fitness function lacks the expressive power of variants. In this paper, we propose the application of a multi-objective genetic algorithm to APR in order to improve efficiency. We also propose a new crossover method that combines two variants with complementary test results, taking advantage of the high expressive power of multi-objective genetic algorithms for evaluation. We tested the effectiveness of the proposed method on competitive programming tasks. The obtained results showed significant differences in the number of successful trials and the required generation time.

There are areas on an application screen where the content changes every time the screen is displayed such as advertisements and news articles (hereafter, dynamic content areas). When there are dynamic content areas, many differences in false positive are detected in Visual Regression Testing. Such differences due to dynamic content areas prevent the tester from efficiently checking the important differences. Therefore, specifying a dynamic content area as a masked area and excluding it from the comparison area is effective for reducing the human effort required to confirm these differences. We propose a method of accurately specifying the area to be masked based on the relative positional relationship of multiple screen elements and excluding dynamic content areas from the comparison area.

Scripted testing and exploratory testing are widely adopted among developers. It is known that exploratory testing can detect bugs more efficiently than scripted testing. However, exploratory testing has the problem that it is difficult to agree with third parties on test results and test quality. In this study, we propose a novel testing approach to facilitate efficient testing and quality assurance. Furthermore, we propose a tool that records the details of testing activity and visualizes them in multiple ways. Our approach realizes both high efficiency like exploratory testing and high auditability like scripted testing. We interviewed 14 developers about our approach to refine it and make it practical. We describe how we reflected the feedback from the interview in our approach.

Programs with interactive and/or realtime activities, such as GUI programs, action game programs, network-based programs, and sensor information processing programs, are not suitable for traditional breakpoint-based debugging, in which execution of the target program is suspended, for two reasons. First, since the timings and order of input event occurrences such as user operations are quite important, such programs do not behave as expected if execution is suspended at a breakpoint. Second, suspending a program to observe its internal states significantly degrades the efficiency of debugging. A debugging method is presented that resolves these problems. It keeps track of both the currently executing statement in a program and the changes in value of expressions of interest, and visualizes them in realtime. The proposed method was implemented as SLDSharp, a debugger for C# programs, by means of a program transformation technique. Through a case study of debugging a practical game program created by using the Unity game engine, it is shown in that SLDSharp makes it possible to efficiently debug.

Test automation tools such as Selenium are commonly used for automating end-to-end tests, but when developers update the software, they often need to modify the test scripts accordingly. However, the costs of modifying these test scripts are a big obstacle to test automation because of the scripts' fragility. In particular, locators in test scripts are prone to change. Some prior methods tried to repair broken locators by using structural clues, but these approaches usually cannot handle radical changes to page layouts. In this paper, we propose a novel approach called COLOR (correct locator recommender) to support repairing broken locators in accordance with software updates. COLOR uses various properties as clues obtained from screens (i.e., attributes, texts, images, and positions). We examined which properties are reliable for recommending locators by examining changes between two release versions of software, and the reliability is adopted as the weight of a property. Our experimental results obtained from four open source web applications show that COLOR can present the correct locator in first place with a 77% - 93% accuracy and is more robust against page layout changes than structure-based approaches.

The regression testing method called Visual Regression Testing checks whether application screens display correctly without any presentation failures. This method uses image comparison tools that detect differences in screen elements, e.g. the disappearance of or change in the position of a button, by comparing images of the correct screen and the target screen by using computer vision techniques. It can detect differences in all the screens of an application automatically, but the tester needs to confirm all the differences detected in the screens. On the other hand, the application screen is often composed of areas such as the header, footer, navigation and content areas. Since the header, footer and navigation are displayed at fixed positions on a screen and commonly exist on many screens, if a difference is detected in them, the same difference will be detected in many other screens. As a result, we need labor to repeat the same confirmation for screens containing the same differences. Here, we propose a method to reduce the labor in making such confirmations by analyzing all the screen comparison results to identify the same differences so that the tester only has to confirm one screen comparison result as a representative. We confirmed the effectiveness of our method on real application screens and confirmed that it reduces the number of differences that the tester needs to confirm by about 60%.

Automated program repair (in short, APR) has been attracting much attention. A variety of APR techniques have been proposed, and they have been evaluated with actual bugs in open source software. Currently, the authors are trying to introduce APR techniques to industrial software development (in short, ISD) to reduce development cost drastically. However, at this moment, there are no studies that report evaluations of APR techniques on ISD. In this paper, we report our ongoing application of APR techniques to ISD and discuss some barriers that we found on the application.

Much work has been done on automating regression testing for applications. But most of them focus on test execution. Little work has been done on automatically determining if a test case passes or fails. This decision is often made by comparing the results of executing test cases on a base version of the application and post-modification version of the application. If the two results match, the test case passes, otherwise fails. However, to the best of our knowledge, there is no regression testing method for automatically deciding pass/fail of dynamic Web applications which use JavaScript or CSS. We propose a method that automatically decides if a dynamic Web application passes a regression test case. The basic idea is to obtain a screenshot each time the GUI of the Web application (i.e. Web page) changes its state, and then compare each corresponding screenshot to see if they match. The evaluation results showed that the accuracy rate of our approach is high and our approach can be considered as fast enough for practical use.

We present CATG, an open-source concolic test generation tool for Java and its integration with TesMa, a model-based testing tool which automatically generates test cases from formal design documents. TesMa takes as input a set of design documents of an application under test. The design documents are provided in the form of database table definitions, process-flow diagrams, and screen definitions. From these design documents, TesMa creates Java programs for the feasible execution scenarios of the application. CATG performs concolic testing on these Java programs to generate suitable databases and test inputs required to test the application under test. A demo video of the tool is available at https://www.youtube.com/watch?v=9lEvPwR7g-Q.

Much work has been done on automating regression testing for Web applications, but most of them focus on test data generation or test execution.Little work has been done on automatically determining if a test passed or failed; testers would need to visually confirm the result which can be a tedious task.The difficulty is compounded by the fact that parts of a Web page (such as advertisements) may change each time the Web application is executed even though it has no bearing on the Web application function itself.We thus propose a test oracle for automatically determining the result of regression testing a Web application.The key point of our approach is the identification of parts that may change, which we call variable region.We first generate the expected result, by executing the original (premodification) Web application multiple times so that variable regions can be identified.Then, after the Web application is modified, regression testing is conducted by comparing the output of the modified Web application against the expected output.An evaluation confirmed the usefulness of our approach.