Continuous Integration (CI) is a fundamental practice in modern software development. It enables early fault detection through regression testing, where Test Case Prioritization (TCP) plays a crucial role in improving detection efficiency. Reinforcement Learning-based TCP (RL-TCP) has shown promise in CI scenarios, but its performance often fluctuates due to CI’s dynamic nature. Existing solutions address this by assigning additional rewards or periodically retraining agents. However, these methods either risk over-adjusting strategies due to excessive reliance on additional rewards or fail to respond promptly to changes due to fixed retraining intervals. We propose a novel Environment Adaptation Agent-based RL-TCP method (EAA) that addresses these challenges through a dual mechanism. EAA detects significant environmental changes by analyzing fluctuations in prioritization effectiveness. When a change is detected, it assigns targeted rewards to test cases. EAA also refines the agent’s gradient update so that environmental dynamics are better incorporated into retraining. This enables agents to swiftly adapt while retaining learned prioritization knowledge. Evaluations on 12 real-world industrial datasets show that EAA improves the NAPFD metric by 4.7–24.79% and reduces the average TTF by 35.85–50.37 positions compared to state-of-the-art RL-TCP methods. Additionally, EAA significantly reduces occurrences of NAPFD equal to zero, effectively mitigating prioritization instability.

Code generation has gained increasing attention as a task to automate software development by transforming high-level descriptions into executable code. While large language models (LLMs) are effective in generating code, their performance heavily relies on the quality of input prompts. Current prompt engineering methods involve manual effort in designing prompts, which can be time-consuming and yield inconsistent results, potentially constraining the efficacy of LLMs in practical applications. This paper introduces Prochemy, a novel approach for automatically refining prompts iteratively to enhance code generation. Prochemy addresses the limitations of manual prompt engineering by automating the optimization process, ensuring prompt consistency during inference, and aligning with multi-agent systems. It iteratively refines prompts based on model performance, using an optimized final prompt to improve consistency and reliability across tasks. We evaluate Prochemy on both natural language-based code generation and code translation tasks using three series of LLMs. Results show that when combining Prochemy with existing approaches, it outperforms baseline prompting methods. It achieves improvements of 5.0% (GPT-3.5-Turbo) and 1.9% (GPT-4o) over zero-shot baselines on HumanEval. For the state-of-the-art LDB, Prochemy + LDB outperforms standalone methods by 1.2–1.8%. For code translation, Prochemy elevates GPT-4o’s performance on Java-to-Python (AVATAR) from 74.5 to 84.1 (+12.9%) and Python-to-Java from 66.8 to 78.2 (+17.1%). Furthermore, considering that the o1-mini model integrates prompt engineering techniques, Prochemy can continue to show good performance among it, further validating its effectiveness in code generation and translation tasks. Additionally, Prochemy is designed to be plug-and-play, optimizing prompts with minimal human intervention and seamlessly bridging the gap between simple prompts and complex frameworks.

Integrating Large Language Models (LLMs) into software fault localization represents a significant advancement in improving debugging efficiency for programmers. However, novice program fault localization, which is essential for computer science education, has not been thoroughly investigated in previous studies. In contrast to industrial programs target practical functionality, novice programs primarily deal with individual algorithmic issues. The distinct logic structures between novice and industrial programs can impact how effectively LLM understand and process them. Moreover, this difference reveals the inapplicability of the Competent Programmer Hypothesis, a fundamental assumption in industrial fault localization, to novice program fault localization. Therefore, industrial methodologies are unsuitable for novice programming, emphasizing the need for our empirical studies. To fill this gap, we evaluate LLMs’ effectiveness in localizing faults for novice programs in statement level. Using the widely used novice programs dataset Codeflaws and Condefects, we compare the performance of two commercial LLMs (i.e., ChatGPT-3.5 and ChatGPT-4) and three open-source LLMs (i.e., ChatGLM3, Llama2, and Code Llama) against traditional fault localization methods, examining their accuracy and overlap. Additionally, we investigate how prompt engineering improves localization precision. Our findings show ChatGPT-4’s overall superior performance, with ChatGPT-3.5 exhibiting minor advantages in certain cases. ChatGPT-4 outperforms the traditional methods with best performance by 592% and 137% on Codeflaws and Condefects. Specifically, each method exhibits unique strengths in localizing novice programming faults. Moreover, carefully crafted prompts can improve LLMs’ precision. These insights underscore the promising potential of utilizing LLMs for fault localization in novice programming.

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is [Formula: see text], [Formula: see text], [Formula: see text] and [Formula: see text] in efficiency, test suite size, diversity and robustness.

Continuous automated testing throughout each cycle can ensure the security of the continuous integration (CI) development lifecycle. Test case prioritization (TCP) is a critical factor in optimizing automated testing, which prioritizes potentially failed test cases and improves the efficiency of automated testing. In CI automated testing, the TCP is a continuous decision-making process that can be solved with reinforcement learning (RL). RL-based CITCP can continuously generate a TCP strategy for each CI development lifecycle, with the reward mechanism as the core. The reward mechanism consists of the reward function and the reward strategy. However, there are new challenges to RL-based CITCP in real-industry CI testing. With high-frequency iteration, the reward function is often calculated with a fixed length of historical information, ignoring the spatial characteristics of the current cycle. Therefore, the dynamic time window (DTW)-based reward function is proposed to perform the reward calculation, which adaptively adjusts the recent historical information range based on the integration cycle. Moreover, with low-failure testing, the reward strategy usually only rewards failure test cases, which creates a sparse reward problem in RL. To address this issue, the similarity-based reward strategy is proposed, which increases the reward objects of some passed test cases, similar to the failure test cases. The DTW-based reward function and the similarity-based reward strategy together constitute the proposed adaptive reward mechanism in RL-based CITCP. To validate the effectiveness of the adaptive reward mechanism, experimental verification is carried out on 13 industrial data sets. The experimental results show that the adaptive reward mechanism can improve the TCP effect, where the average NAPFD is maximally improved by 7.29%, the average Recall is maximally improved by 6.04% and the average TTF is improved by 6.81 positions with a maximum of 63.77.

Abstract Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

MBFL (Mutation-Based Fault Localization) is one of the most commonly studied fault localization techniques due to its promising fault localization effectiveness. However, MBFL incurs a high execution cost as it needs to execute the test suite on a large number of mutants. While previous studies have proposed mutant reduction methods for FOMs (First-Order Mutants) to help alleviate the cost of MBFL, the reduction of HOMs (Higher-Order Mutants) has not been thoroughly investigated. In this study, we propose SGS (Statement Granularity Sampling), a method which conducts HOMs reduction for HMBFL (Higher-Order Mutation-Based Fault Localization). Considering the relationship between HOMs and statements, we sample HOMs at the statement level to ensure each statement has corresponding HOMs. We empirically evaluate the fault localization effectiveness of HMBFL using SGS on 237 multiple-fault programs taken from the SIR and Codeflaws benchmarks. The experimental results show that (1) The best sampling ratio for HMBFL with SGS is 20%, which preserves the performance and reduces execution costs by 80% ; (2) The fault localization accuracy of HMBFL with SGS outperforms the state-of-the-art SBFL (Spectrum-Based Fault Localization) and MBFL techniques by 20%.

Test case prioritization based on reinforcement learning has been seen as a promising way to achieve continuous integration testing. Agent and reward function are two crucial components of reinforcement learning. During the process of reinforcement learning in continuous integration test case prioritization, the agent decides on the execution order of test cases (actions) for the next integration testing (environment) based on the corresponding test case features (states), aiming to detect errors early by maximizing the reward. Furthermore, having more test case features allows the agent to perceive the environment better, but it also increases computation consumption and brings convergence problems to learning. In this paper, we first propose a multi-feature environment perception for continuous integration test case prioritization. It introduces multiple features based on test case history execution information to solve the agent’s limitation in obtaining environmental information. Additionally, we propose a self-attention agent network structure, which captures relationships between multiple features to prevent the convergence problem of reinforcement learning. An extensive experimental and analytical study was conducted with 15 existing reward functions on 14 industrial data sets. The results show that (1) the proposed multiple features can help the agent to perceive environmental information more comprehensively, and (2) the proposed self-attention agent can process environmental information better to achieve more effective test case prioritization in continuous integration testing.

Spectrum-Based Fault Localization (SBFL) technique is widely applied for fault localization, identifying faulty statements potentially resulting in unexpected faulty programs’ behavior. However, researchers have approved that Coincidental Correct (CC) test cases contained in test suites can negatively affect the accuracy of SBFL. Previous researchers sought to identify CC test cases through machine learning algorithms, but the feature representation is insufficient, leading to limited accuracy. To address this challenge, we propose the Machine Learning-based CC test cases Identification approach (MLCCI), which leverages multiple features extracted from the program under test to identify CC test cases and map the CC identification task to a learning problem. To evaluate the performance of MLCCI, we conduct experiments in the well-known dataset Defects4J. The experimental results compared with state-of-the-art baselines indicate that: (1) MLCCI achieves higher CC identifying accuracy, with the average Recall, P recision, and F -measure values of MLCCI are 65.93%, 71.69%, and 53.74%, respectively; (2) The fault localization accuracy of MLCCI with the Jaccard formula outperforms baselines, where the values of Accuracy@ 1, 3, and 5 are 347, 369, and 393, achieving the maximum 137.67%, 67.73%, and 47.74% improvement against baselines, respectively. Besides, we perform ablation analysis to reveal the effectiveness of features utilized in this study.

Mutation-based fault localization has shown promising accuracy in localizing faults due to its finer analysis granularity. However, the effectiveness is limited when dealing with diverse real-world systems and actual faults, which results from its inflexible suspiciousness calculation and oversimplification of information. In this work, we propose a novel Mutation-Based Fault Localization technique, GMBFL, which utilizes Graph representation to achieve multi-information cooperation to improve fault localization. GMBFL comprises two key components: a fine-grained graph-based representation to fully utilize the information of the program, and an effective suspiciousness measure using the graph neural network to learn useful features from the graph. We evaluate GMBFL on 243 real faulty programs from Defects4J. The experimental results show that GMBFL can surpass both the state-of-the-art learning-based fault localization technique and 70 commonly used SBFL and MBFL techniques. In particular, GMBFL localizes 125 faults within TOP-1 whereas the best baseline technique can at most localize 109 faults within TOP-1.

Code comments are essential for software development and maintenance, as they provide natural language descriptions of the code that help developers understand the program and reduce the time spent on comprehension. However, writing code comments can be tedious and time-consuming, and many software projects lack comprehensive and up-to-date comments, which can impair the readability and maintainability of programs.

Automatically localizing faulty statements is a desired feature for effective learning programming. Most of the existing automated fault localization techniques are developed and evaluated on commercial or well-known open-source projects, which performed poorly on novice programs. In this paper, we propose a novel fault localization technique VsusFL (Variable-suspiciousness-based Fault Localization) for novice programs. VsusFL is inspired by simulating the manual program debugging process and takes advantage of variable value sequences. VsusFL can trace variable value changes, determine whether the intermediate state of the variables is correct, and report the potential faulty statements for novice programs. This paper presents the implementation of VsusFL and conducts empirical studies on 422 real faulty novice programs. Experimental results show that VsusFL performs much better than Grace, ANGELINA, VSBFL, Spectrum-Based Fault Localization (SBFL), and Variable-based Fault Localization (VFL) in terms of T O P -1, T O P -3, and T O P -5 metrics. Specifically, VsusFL can localize 90%, 35% and 9% more faulty statements than the best-performing baseline Grace. Moreover, We analyze the correlation between VsusFL and other techniques and find a weak correlation since they perform well on different programs, indicating the potential to further enhance fault localization performance through strategic integration of VsusFL with other methods.

Abstract Compilation errors are unavoidable during the debugging process of novice students. Compiler error messages can help novices to localize and remove errors, but these messages are difficult to understand for students. Previous studies have investigated the compilation error categorization by analyzing compiler error messages, but the categorization cannot cover all kinds of errors, which limits the evaluation of compilation error studies. Therefore, a comprehensive categorization for compilation errors is needed for evaluating the performance of models or tools related to compilation errors. In this study, we first propose a new compilation error categorization, which is based on the smallest unit of the program, tokens. The experiments on 29,573 programs from three datasets show that our proposed compilation error categorization can cover more types of errors and the distribution of the error categorization are significantly different between the datasets. Then, based on our proposed categorization, we develop a neural network model CLACER (CLAssification of Compilation ERrors) for predicting the compilation errors. The results indicate that CLACER can improve the compiler's error localization accuracy and predicts the compilation error effectively. Moreover, based on the proposed categorization, we conduct empirical studies to evaluate the performance of three repairing tools (i.e., DeepFix, RLAssist, and MACER). The comparison results illustrate that DeepFix and RLAssist can fix more errors in the category of delimiter than errors in other categories. Furthermore, MACER performs better than DeepFix and RLAssist because it has a sufficient repairing pattern set for the errors. We also provide some suggestions for improving the repairing tools in the future.

Abstract Reinforcement learning (RL) has been used to optimize the continuous integration (CI) testing, where the reward plays a key role in directing the adjustment of the test case prioritization (TCP) strategy. In CI testing, the frequency of integration is usually very high, while the failure rate of test cases is low. Consequently, RL will get scarce rewards in CI testing, which may lead to low learning efficiency of RL and even difficulty in convergence. This paper introduces three rewards to tackle the issue of sparse rewards of RL in CI testing. First, the historical failure density‐based reward (HFD) is defined, which objectively represents the sparse reward problem. Second, the average failure position‐based reward (AFP) is proposed to increase the reward value and reduce the impact of sparse rewards. Furthermore, a technique based on additional reward is proposed, which extracts the test occurrence frequency of passed test cases for additional rewards. Empirical studies are conducted on 14 real industry data sets. The experiment results are promising, especially the reward with additional reward can improve NAPFD (Normalized Average Percentage of Faults Detected) by up to 21.97%, enhance Recall with a maximum of 21.87%, and increase TTF (Test to Fail) by an average of 9.99 positions.

Abstract Identifying the location of faults in real‐world programs is one of the costly processes during software debugging. To reduce the debugging effort, various fault localization techniques have been proposed in recent years. Spectrum‐based fault localization (SBFL) is one kind of widely investigated fault localization technique. Most SBFL techniques first calculate the suspiciousness of program elements (such as statements, methods) to be faulty using the coverage information and execution results of tests. Then a rank list of program elements is generated according to their suspiciousness. However, some SBFL techniques only consider the binary coverage information (i.e., whether the program element is covered) but ignore some of the tests' running behaviors, such as the execution frequency when faults occur in the iteration entities or loop bodies, which are more likely to be faulty followed the propagation‐infection‐execution model. The execution frequency based techniques only replace the feature items of the existing formula limiting their effectiveness in fault localization. In this article, we propose a fault localization technique, class reduction and method call frequency (CRMF), which utilizes mutation analysis and information retrieval techniques. In particular, CRMF first uses mutation analysis to identify and reduce the classes, in which the program elements with a low probability of being faulty. Then we propose a new suspiciousness formula that applies information retrieval and considers method call frequency. To evaluate the effectiveness of CRMF, we conduct empirical studies on 264 real‐world programs from the Defects4J benchmark. Final results show that CRMF outperforms the statement frequency based technique FLSF and SBFL techniques (i.e., Ochiai, OP2, Tarantula, and Dstar) in both single‐fault programs and multiple‐fault programs. Specifically, CRMF can rank 29, 74, and 112 faults at the top 1, 3, 5 ranks and achieve a higher mean reciprocal rank for single‐fault programs and multiple‐fault programs. Finally, we discuss the essence of CRMF and analyze its effectiveness on multi‐fault programs in detail.

Summary Heuristic algorithms are widely used to solve multi‐objective test case prioritization (MOTCP) problems. However, they perform differently for different test scenarios, which conducts difficulty in applying a suitable algorithm for new test requests in the industry. A concrete hyper‐heuristic framework for MOTCP (HH‐MOTCP) is proposed for addressing this problem. It mainly has two parts: low‐level encapsulating various algorithms and hyper‐level including an evaluation and selection mechanism that dynamically selects low‐level algorithms. This framework performs good but still difficult to keep in the best three. If the evaluation mechanism can more accurately analyse the current results, it will help the selection strategy to find more conducive algorithms for evolution. Meanwhile, if the selection strategy can find a more suitable algorithm for the next generation, the performance of the HH‐MOTCP framework will be better. In this paper, we first propose new strategies for evaluating the current generation results, then perform an extensive study on the selection strategies which decide the heuristic algorithm for the next generation. Experimental results show that the new evaluation and selection strategies proposed in this paper can make the HH‐MOTCP framework more effective and efficient, which makes it almost the best two except for one test object and ahead in about 40% of all test objects.

Automated code comment generation technologies can help developers understand code intent, which can significantly reduce the cost of software maintenance and revision. The latest studies in this field mainly depend on deep neural networks, such as convolutional neural networks and recurrent neural network. However, these methods may not generate high-quality and readable code comments due to the long-term dependence problem, which means that the code blocks used to summarize information are far from each other. Owing to the long-term dependence problem, these methods forget the previous input data’s feature information during the training process. In this article, to solve the long-term dependence problem and extract both the text and structure information from the program code, we propose a novel improved-Transformer-based comment generation method, named SeTransformer. Specifically, the SeTransformer utilizes the code tokens and an abstract syntax tree (AST) of programs to extract information as the inputs, and then, it leverages the self-attention mechanism to analyze the text and structural features of code simultaneously. Experimental results based on public corpus gathered from large-scale open-source projects show that our method can significantly outperform five state-of-the-art baselines (such as Hybrid-DeepCom and AST-attendgru). Furthermore, we also conduct a questionnaire survey for developers, and the results show that the SeTransformer can generate higher quality comments than those of other baselines.

In software regression testing, newly added test cases are more likely to fail, and therefore, should be prioritized for execution. In software regression testing for continuous integration, reinforcement learning-based approaches are promising and the RETECS (Reinforced Test Case Prioritization and Selection) framework is a successful application case. RETECS uses an agent composed of a neural network to predict the priority of test cases, and the agent needs to learn from historical information to make improvements. However, the newly added test cases have no historical execution information, thus using RETECS to predict their priority is more like ‘random’. In this paper, we focus on new test cases for continuous integration testing, and on the basis of the RETECS framework, we first propose a priority assignment method for new test cases to ensure that they can be executed first. Secondly, continuous integration is a fast iterative integration method where new test cases have strong fault detection capability within the latest periods. Therefore, we further propose an additional reward method for new test cases. Finally, based on the full lifecycle management, the ‘new’ additional rewards need to be terminated within a certain period, and this paper implements an empirical study. We conducted 30 iterations of the experiment on 12 datasets and our best results were 19.24%, 10.67%, and 34.05 positions better compared to the best parameter combination in RETECS for the NAPFD (Normalized Average Percentage of Faults Detected), RECALL and TTF (Test to Fail) metrics, respectively.

Fault localization is one of the most expensive activities in software debugging. Mutation-based fault localization (MBFL) is a commonly studied technique that applied mutation analysis to find the location of faults in the programs. Previous studies showed that MBFL adopted First-Order-Mutants (FOMs) that could achieve promising results in single-fault localization, but it did not perform well in multiple-fault localization. Recently, Higher-Order-Mutants (HOMs) were proposed for modeling complex faults but whether HOMs can help in fault localization is still unknown. In this paper, we investigate the performance of MBFL with FOMs and HOMs on single- and multiple-fault localization. Moreover, to study the characteristics of HOMs, we divide HOMs into three groups (i.e. Accurate HOMs, Partially accurate HOMs, and Inaccurate HOMs) by considering different mutation locations. Based on the empirical results on 186 versions of six real-world programs, we find that (1) In single-fault localization, FOMs can achieve better performance than HOMs. (2) However, in multiple-fault localization, HOMs (2-HOMs) localize more faults than FOMs. (3) Furthermore, different types of HOMs have different fault localization effectiveness, where Accurate HOMs outperform the other two HOMs categories. Therefore, the researchers should propose methods to find HOMs more useful for fault localization.

First-order mutants (FOMs) have been widely used in mutation-based fault localization (MBFL) approaches and have achieved promising results in single-fault localization scenarios (SFL-scenario). Higher-order mutants (HOMs) are proposed to simulate complex faults and can be applied in MBFL theoretically for multiple-fault localization scenarios (MFL-scenario). However, whether HOMs can improve MBFL’s performance is not investigated and the effectiveness is not thoroughly evaluated. In this empirical study, we investigate the impact of HOMs on the performance of MBFL in SFL-scenario and MFL-scenario. The experiments on two real-world benchmarks reveal that 1) 2-HOMs can help improve the MBFL performance in SFL-scenarios; 2) in MFL-scenarios, both 2-HOMs and 3-HOMs can achieve better performance than FOMs; and 3) huge computational cost cannot be ignored in the practice of HOMs. Therefore, effective methods to reduce the number of HOMs for future MBFL studies should be considered.

To improve the efficiency of the fault localization process, different automatic fault localization approaches have been proposed. Among these approaches, the spectrum-based fault localization (SBFL) approach has been widely used and studied due to its lightweight and high effectiveness. However, while the existence of coincidental correct (CC) test cases can influence the usefulness of SBFL in single-fault programs, their influence on multiple fault programs has not been thoroughly investigated. Therefore, in this article, we conduct a theoretical analysis and an empirical study to investigate the effect of CC test cases on multiple fault localization. The theoretical analysis is based on a suspiciousness calculation formula of SBFL, which divides CC test cases into three categories (specific, irrelevant, and unspecific) according to their association with a specific faulty statement. Following this analysis, we conduct an empirical study on two well-known open-source repositories (SIR and Defects4J), and the experimental results verify the correctness of our theoretical analysis. Specifically, reducing the number of specific CC test cases for a faulty statement can improve or maintain fault localization accuracy, while eliminating irrelevant CC test cases can have a negative effect. Finally, we design a CC test case identification solution based on the isolation-based multiple fault localization approach and demonstrate its effectiveness via a simulation experiment.

Compilation errors happen during the debugging process of novice students. Compiler error messages help novices to localize and remove errors, but these messages are difficult to understand for students. Some computing education researchers analyzed the compiler error messages generated by novice’s attempts to compile their programs. However, some important questions remain open. For example, the existing compilation error category cannot cover all programs due to the simple static analysis and program repair patterns. And existing prediction models for classifying compilation errors are unsatisfactory because of the inappropriate neural networks. In this paper, we first propose a new category of compilation error based on the program tokens, which is the smallest unit of the program. Then we develop a neural network model CLACER (ClAssification of Compilation ERrors) based on TextCNN. CLACER performs better on extracting semantic features and statistical features from compiler error messages. To verify the effectiveness of our proposed category and corresponding method CLACER, we choose 16,926 student programs as our experimental subjects. Final experimental results indicate that our proposed classification category covers 16.5% more programs than the state-of-the-art category TEGCER. Moreover, CLACER improves the compiler’s localization effectiveness and with a 4.25% improvement on the TEGCER category. Further analysis shows that CLACER has a promising prediction performance for different error classes, and TextCNN is more suitable for constructing the compilation error classification model.

Test Case Prioritization (TCP) based on the continuous decision of Reinforcement Learning (RL) has achieved a successful application for test cases optimization in Continuous Integration (CI). The reward functions of RL describe how a test case "ought" to be executed in next integration, of which the design is usually based on the historical executions of the test case. The Average Percentage of Historical Failure (APHF) had been considered as one of the best reward function which has a strong correlation with the recent failure executions of a test case. However, for a test case with many historical failures but passes in recent cycles, the APHF value may be low. In this paper, two novel reward functions are proposed focusing on the impact of failure position in test case history execution sequence, which are the Average Position Exponential Weight (APEW) reward function and the Average Position Quadratic Weight (APQW) reward function, respectively. Both APEW and APQW carry out weight design of failure position but with different weights. We theoretically prove the issue of the only strong correlation with recent failure executions, and also prove that both proposed reward functions can reflect the quantity of historical failures and the distribution of these failures. Experimental verification on 10 industrial-level data sets show that the proposed reward functions can effectively improve the fault detection capability of test cases.

In the continuous integration, test case prioritization can effectively alleviate the resource-intensive problems associated with frequent integration commits. Test case prioritization in continuous integration is a sequential decision problem from which reinforcement learning is applied and can effectively adapt and learn from a changing environment. However, continuous integration testing brings new problems of sparse rewards to reinforcement learning because of frequent integration with low test failure and this problem can be addressed by increasing the number of rewarded test cases. In this paper, we propose a reinforcement learning reward object selection strategy based on Test Case Synchronization and Diversity (TCSD) that rewards failed test cases and with an additional selection of passed test cases with potential failure ability. The experiments on six real-world industrial data sets show that TCSD improves the learning efficiency and fault detection ability of reinforcement learning 6.35% in average NAPFD compared with the traditional strategies.

Identifying the location of faults in real-world programs is one of the most costly processes during software debugging. In order to reduce debugging effort, many fault localization techniques have been proposed. One of the most widely studied technique is called Spectrum-based fault localization (SBFL), which uses the coverage information and execution results of test cases to do fault localization. Most SBFL techniques only consider the binary coverage information and ignore the execution frequency, so their fault localization accuracy is limited, especially when faults occur in the iteration entities or loop bodies. In this paper, we propose IRBFL, a novel fault localization technique based on information retrieval to extract information from execution frequencies of program entities. IRBFL uses mutation analysis to reduce the low suspicious classes, and then it adopts information retrieval techniques to calculate the suspiciousness value. We evaluate IRBFL on 205 real-world faults from 5 programs in Defects4J benchmark. The experimental results show that our proposed method outperforms the other five state-of-the-art SBFL techniques. More specifically, no matter in single-fault or multi-fault programs, IRBFL can identify 2 to 3 times more faulty methods than the other five SBFL techniques when checking the top 1 method. More empirical results in terms of other metrics, including acc@3, acc@5, EXAM, MRR, and MAP, also indicate that IRBFL technique is better than the other five SBFL techniques.

By scheduling algorithms in the low-level algorithm library, the hyper-heuristic algorithm can help to effectively select an appropriate method to deal with hard computational search problems. The hyper-heuristic algorithm usually includes a high-level scheduling layer and a low-level algorithm layer. The high-level strategy layer selects the algorithm for the next scheduling by evaluating the execution effect of the different algorithms in the low-level layer, while the low-level layer includes a variety of different heuristic algorithms which called algorithm library. The concrete hyper-heuristic framework for multi-objective test case prioritization was presented where the 18 multi-objective algorithms were formed in the low-level library. It has been gradually realized that a hybrid algorithm by combining single objective algorithm and multi-objective optimization algorithm is better than the individual. This paper explores the influence of the construction pattern of algorithm library for the hyper-heuristic algorithm by constructing the fusion pattern of different types of algorithms.

Online Judge (OJ) system, which can automatically evaluate the results (right or wrong) of programs by executing them on standard test cases, is widely used in programming education. While an OJ system with personalized feedback can not only give execution results, but also provide information to assist students in locating their problems quickly. Automatically fault localization techniques are designed to find the exact faults in programs automatically, experimental results showed their effect on locating artificial faults, but their effectiveness on novice programs needs to be investigated. In this paper, we first evaluate the effectiveness of several widely-studied fault localization techniques on novice programs, and then we use fine-grained test cases to improve the fault localization accuracy. Empirical studies are conducted on 77 real student programs and the results show that, compared with original test cases in OJ system, the fault localization accuracy can be improved obviously when using fine-grained test cases. More specifically, in terms of TOP-1, TOP-3 and TOP-5 metrics, the maximum results can be improved from 5, 22, 37 to 9, 24, 48, respectively. The results indicate that more faults can be located when checking the top 1, 3 or 5 statements, so the fault localization accuracy is enhanced. Furthermore, a Test Case Granularity (TCG) concept is introduced to describe fine-grained test cases, and empirically studies demonstrate that there is a strong correlation between TCG and fault localization accuracy.

Continuous Integration (CI) testing is an expensive, time-consuming, and resource-intensive process. Test case prioritization (TCP) can effectively reduce the workload of regression testing in the CI environment, where Reinforcement Learning (RL) is adopted to prioritize test cases, since the TCP in CI testing can be formulated as a sequential decision-making problem, which can be solved by RL effectively. A useful reward function is a crucial component in the construction of the CI system and a critical factor in determining RL’s learning performance in CI testing. This paper focused on the validity of the execution history information of the test cases on the TCP performance in the existing CI testing optimization methods based on RL, and a Dynamic Time Window based reward function are proposed by using partial information dynamically for fast feedback and cost reduction. Experimental studies are carried out on six industrial datasets. The experimental results showed that using dynamic time window based reward function can significantly improve the learning efficiency of RL and the fault detection ability when comparing with the reward function based on fixed time window.

In continuous integration (CI) environments, the program is rapidly and frequently modified and integrated. This feature introduces significant challenges to testing processes conducted in these environments. Based on existing technology, a test case that fails frequently is likely to fail in future tests. Therefore, the historical execution results of test cases are essential to guide the test case prioritization (TCP) in the CI environment. Reinforcement learning involves solving sequential decision-making problems and is suitable for TCP in the CI environment. At present, most of the TCP techniques based on reinforcement learning rely on the current cycle historical failure information of test cases. They rarely consider more historical cycle information, as well as other influencing factors. In this paper, we discussed the occurrence frequency of test cases for the first time. We also considered all historical information of each test case and proposed three new reward function, which employs the percentage of historical failure and the failure distribution of test cases, which can guide the reinforcement learning process. We evaluate our method on five industrial data sets. The experimental results show that our method can effectively prioritize test cases and improve the cost-effectiveness of the CI process.

Learning agent plays significant role in the hyper-heuristic framework for test case prioritization, where an evaluation strategy is applied to evaluate the execution results produced by the current heuristic algorithm and select the most appropriate heuristic algorithm for the next generation. Hierarchical Distribution (HD) is used as evaluation strategy based on the dominance relationship between the individuals from the present and last generations. In addition to the distribution of the solution set, a good convergence towards the optimal Pareto front is often desired. In this paper, the convergence ability of the individuals is further considered in the design of the evaluation strategy for the learning agent, in which Pareto Dominance and Convergence Information are adopted. Three evaluation strategies are proposed and empirically studied, and the experimental results show that the hyper-heuristic algorithms with the proposed evaluation strategies are more effective and efficient for test case prioritization.

Testing multi-thread programs becomes extremely difficult because thread interleavings are uncertain, which may cause a program getting different results in each execution. Thus, Thread Scheduling Sequence (TSS) is a crucial factor in multi-thread program testing. A good TSS can obtain better testing efficiency and save the testing cost especially with the increase of thread numbers. Focusing on the above problem, in this paper, we discuss a kind of approach that can efficiently generate TSS based on the concurrent coverage criteria. First, we give a definition of Synchronization Pair (SP) as well as all Synchronization Pairs Coverage (ASPC) criterion. Then, we introduce the Synchronization Pair Thread Graph (SPTG) to describe the relationships between SPs and threads. Moreover, this paper presents a TSS generation method based on the ASPC according to SPTG. Finally, TSSs automatic generation experiments are conducted on six multi-thread programs in Java Library with the help of Java Path Finder (JPF) tool. The experimental results illustrate that our method not only generates TSSs to cover all SPs but also requires less state number, transition number as well as TSS number when satisfying ASPC, compared with other three widely used TSS generation methods. As a result, it is clear that the efficiency of TSS generation is obviously improved.

In this article, test diversity has been suggested to be a valid way to improve test suite effectiveness. Extended finite state machine (EFSM) is a widely used formal model, but little attention is paid on the test suite generation with more diversity. EFSM test suite generation involves test paths generation and test data generation. Considering the discrepancy between test paths has a more crucial impact on the diversity of test suite, compared with the difference between test data, this article, therefore, mainly concerns the test paths generation with more diversity for EFSM models. Hence, the factors that influence the discrepancy between test paths are investigated. Then based on these factors, an integrated distance metric is designed to evaluate the dissimilarity between test paths, and a diversity measurement for EFSM test suite is presented. Furthermore, a diversity-oriented test suite generation (DOTSG) method is proposed where a dissimilarity-based fitness function and diversity-oriented update strategy are adopted in traditional coverage-oriented EFSM test suite generation (COTSG) by genetic algorithm. The experimental results show that, compared to COTSG, our DOTSG can not only generate more diverse test suite to satisfy a certain coverage criteria, improving the fault detection capability of the test suite, but also decrease the evolution time cost and the size of test suite generated.

The imbalanced nature of class in software defect data, which including intra-class imbalance and inter-classes imbalance, increases the difficulty of learning an effective defect prediction model. Most of sampling and example generation approaches just focused on inter-class imbalanced defect data, and they are not effective to handle the issue of intra-class imbalance. This paper proposed a distribution based data generation approach for software defect prediction to deal with inter-class and intra-class imbalanced data simultaneously. First, the classified sub-regions are clustered according to the distribution in the sample feature space. Second, the data are generated by corresponding strategies according to different distribution in sub-regions, where the inter-class balance is achieved by increasing the number of defective samples, and the intra-class balance is achieved by generating different density of data in different sub-regions. Experiment results show that the proposed method can reduce the impact of data imbalance on defect prediction and improve the accuracy of software defect prediction model effectively by generating inter-class and intra-class balanced defects data.

Continuous integration refers to the practice of merging the working copies of all developers into the mainline frequently. Regression testing for each mergence is characterized by continually changing test suite, limited execution time, and fast feedback, which demands new test optimization techniques. Reinforcement learning is introduced for test case prioritization to save computing resources in continuous integration environment, where a reasonable reward function is highly important for learning strategy, since the process of reinforcement learning is a reward-guided behavior. In this paper, APHFW, a novel reward function is proposed by using partial historical information of test cases effectively for fast feedback and cost reduction. The experiments are based on three open-source data sets, and the results show that the proposed reward function is more cost-effect than other reinforcement learning rewards in continuous integration environment.

Currently, more than 90% web applications are potentially vulnerable to attacks from both the client side and server side. Test case generation plays a crucial role in testing web applications, where most existing studies focus on test case generation either from client-side or from server-side to detect vulnerabilities, regardless of the interactions between client and server. Consequently, it is difficult for those test cases to discover certain faults which involve both client and server. In this paper, the server-side sensitive paths are considered as vulnerable code paths due to insufficient or erroneous filtering mechanisms. An evolutionary testing approach based on the memetic algorithm is proposed to connect the server-side and client-side, in which test cases are generated from the client-side behavior model, while guided by the coverage of sensitive paths from server-side. The experiments are conducted on four open source web applications, and the results demonstrate that our approach can generate test cases from the client-side behavior model that can cover the server-side sensitive paths, on which the vulnerabilities can be detected more effectively.

Bug isolation is the main approach to multi-fault localization, where failed test cases are divided into groups, and each group failed test cases are used to localize a single fault combined with all passed test cases. Ideally, all failed test cases within a single group execute the same faulty statements. However, misgrouping usually occurs due to the clustering algorithms may not able to divide failed test cases accurately. This paper focuses on the impact of fault localization by the accuracy of the clustering algorithm. A large quantitative empirical study is conducted on 12786 version programs with multiple faults, in which the misgrouping are simulated with different accuracy by a controlled experiment. The results indicate that the effect of fault localization will become worse as the accuracy of clustering decreases.

Test code generated by using the page object design pattern during web testing is easy to maintain.Page clustering is an essential stage of the page object approach.However, existing methods only consider the DOM structure in page clustering, which leads to inaccuracy when generating page objects.A state with the same DOM structure may result in an entirely different migration.The method of considering only the DOM structure cannot accurately generate page object classes.In order to improve the accuracy of page object generation, this paper not only considers DOM structure information but also considers CSS styles and the attributes of DOM elements in page clustering.Based on the experimental evaluation results, our method can automatically generate page objects that cover most of the application functions, which is more effective for the creation and maintenance of web test cases.

Program faults are unavoidable phenomena in the software development. The application of mutation and fault localization techniques is effective in automated program repair, but suffers the inevitable high execution cost as the number of mutants increases exponentially for large industrial programs. The combination with fault localization techniques can reduce the cost by mutating the statements with high suspicious values first. However, the accuracy of fault localization techniques has not been good enough for real applications, and some faults may occur on a statement which is related to the statement with a high suspicious value rather than itself. Therefore, the greedy strategy used currently may not be effective, resulting in inefficient repairs. Finding a mutant as a correct patch should be regarded as a continuous process of searching the global solutions. In this paper, we proposed the search-based automated program repair using mutation and fault localization, which not only takes advantage of fault localization but also overcomes the disadvantage of the greedy strategy used in the mutation generation. The initial population of the search algorithm is constructed by the mutants generated from the statements with high suspicious values using the fault localization, which is a set of rough solutions. A hybrid-crossover operator is then designed where the fixed position crossover operator is used to converge to the global optimal solutions and the random position crossover operator is used to explore the entire search space faster, respectively. The experimental results on the Siemens suite indicate that the proposed approach can improve the efficiency with the same effectiveness compared to the exhaustive approach, and show that the non-random initial population method and the hybrid-crossover strategy can improve the efficiency of the search process.

Most of web applications models focus on sequencing of events, where the ignored parameters or DOM elements changes and the relationship between the execution conditions and web states are crucial for analyzing and testing the behavior of client-side of client-server web applications. In this paper, we first define a novel trace, which can represent dynamic behaviors of web applications more accurately. Then an EFSM-oriented minimal traces set generation approach is proposed for modelling web applications. In order to ensure the integrity of the EFSM model and improve the effectiveness of the modelling process, three adequacy criteria with respect to all events, JS branches and DOM structures, are applied to compensate the traces and to guide the minimal traces set generation by greedy algorithm. Finally, the minimal traces set is abstracted into an EFSM as the behavior model for web applications. We implement a prototype tool for the proposed approach and empirically evaluate that the minimal traces set generation approach based on two web applications. The results show that the traces generated by the approach is effective and the all JavaScript branches coverage criteria is most appropriate to select the minimal traces set used for modelling.

Test cases generation is a principal process in web application testing.Most existing methods generate test cases for improving test efficiency mainly from the aspects like minimizing the test case suite, increasing the code coverage, and so on.However, similar with traditional software having important functions, classes or modules, some web states are more vital than others in web applications.It can be thought that those vital web states relatively have higher influence on the performance of web application.So, they should be given more attention in test case generation.In more detail, the importance of web states can be measured from its page contents or topological structures.Meanwhile, as we known Model-based Testing is a kind of widely used approach in automatic test case generation.Therefore in this paper we propose an EFSM based test case generation method considering the importance of web states for web applications.The experimental results show that our methods can deterministically enhance the testing efficiency of web application.

Abstract Test case prioritization (TCP), which aims to find the optimal test case execution sequences for specific testing objects, has been widely used in regression testing. A wide variety of search methodologies and algorithms have been proposed to optimize test case execution sequences, namely, search‐based TCP. However, different algorithms perform differently and have different implementation costs and specific situations where an algorithm usually performs with high effectiveness and efficiency. When facing a new testing scenario, it is actually difficult to decide which algorithm is suitable. In this paper, to address the algorithm selection problem for different test scenarios, a more generally applicable algorithm based on a hyperheuristic strategy is proposed for search‐based TCP. This includes a range of multiobjective algorithms with a variety of crossover strategies and a learning agent strategy to evaluate and select the appropriate algorithm execution sequence dynamically for different scenarios. The concrete hyperheuristic framework for multiobjective TCP is presented with an algorithm's repository in the low level and the learning agent strategy in the higher level. Experiments show that the proposed learning agent strategy can accurately evaluate algorithms in multiobjective problems and select the appropriate algorithm in each iteration.

The commonly-used software fault localization approaches mainly utilize test coverage information and test cases execution results to calculate the suspiciousness of each program entity to identify the location of faults, namely spectrum based software fault localization (SBFL). It had been argued that such techniques are not helpful in real debugging process, since the low accuracy of localization and few information provided to programmers. In this paper we consider the combination of statement based fault classification with the SBFL, aiming at increasing accuracy of fault localization and provide additional possible fault information to programmers. An improved technique, fault classification oriented SBFL (FC-SBFL), is proposed in this paper, in which the suspiciousness value is adjusted dynamically based on the probability of statement being faulty. Experimental results on real application programs show that FC-SBFL is more effective than SBFL to locate faults, and studies with Tarantula and OP2 show that more than 75% faults have been identified in a better effectiveness.

Mutation Based Fault Localization(MBFL) is a fault localization technique based on mutation analysis, which precisely identifies the location of fault but incurs a high execution cost, since it needs to execute the test suite on a large amount of mutants. Reduction strategies proposed are usually regarding selecting mutation operators or sampling mutants directly, meanwhile at the cost of losing precision of fault localization. This paper proposes a Statement-Oriented Mutant Reduction strategy (SOME), which selects a proportion of mutants at the statement level, specifically, the statements covered by failed tests. SOME keeps the advantage of using whole types of mutation operators, and further considers the increase of mutants' diversity to avoid the precision loss of fault localization. Empirical studies are conducted on 112 faulty versions from 7 benchmark programs, and the results indicate that SOME can reduce 73.51%-79.98% mutation execution cost while keeping almost the same fault location precision as the original MBFL without reduction..

Test data generation is a process of creating program inputs that satisfy specific testing criteria. Many works have been focused on test generation with respect to numeric and string data. Dynamic data structures, such as trees and linked lists, have been widely used in modern programming, but on which there are few studies presented. In general, generating a dynamic data structure is associated with a proper shape and valid values generation. It would be difficult to generate such dynamic data structures, as both shapes and values are necessary to be valid simultaneously. This paper focuses on binary tree structures and proposes a novel test generation approach that combines search based testing with constraint solving techniques. The approach creates the shapes of binary tree structures by using GA, and generates the values in their data fields by using constraint solving techniques. The experimental results show that the presented approach is promising and effective. Moreover, the studies investigate factors affecting the performance of the approach, and arrive at a conclusion that the test generation cost is cubic growing as the number of pointer constraints increases.

Several authors have found evidence of large dependence clusters in the source code of a diverse range of systems, domains, and programming languages. This raises the question of how we might efficiently locate the fragments of code that give rise to large dependence clusters. We introduce an algorithm for the identification of linchpin vertices, which hold together large dependence clusters, and prove correctness properties for the algorithm’s primary innovations. We also report the results of an empirical study concerning the reduction in analysis time that our algorithm yields over its predecessor using a collection of 38 programs containing almost half a million lines of code. Our empirical findings indicate improvements of almost two orders of magnitude, making it possible to process larger programs for which it would have previously been impractical.

Slicing is useful for many software engineering applications and has been widely studied for three decades, but there has been comparatively little work on slicing extended finite state machines (EFSMs). This paper introduces a set of dependence-based EFSM slicing algorithms and an accompanying tool. We demonstrate that our algorithms are suitable for dependence-based slicing. We use our tool to conduct experiments on 10 EFSMs, including benchmarks and industrial EFSMs. Ours is the first empirical study of dependence-based program slicing for EFSMs. Compared to the only previously published dependence-based algorithm, our average slice is smaller 40 percent of the time and larger only 10 percent of the time, with an average slice size of 35 percent for termination insensitive slicing.

Search techniques can dramatically change our ability to solve a host of problems in applied science and engineering, many search techniques have been developed and applied successfully in many fields, including search based software engineering (SBSE). As a key problem of combinatorial testing, covering array generation has been widely studied and many search techniques have been applied which can be named as search based combinatorial testing (SBCT). SBCT is a branch of search based software testing (SBST) within SBSE. In this paper, to explore the applicability and effectiveness of SBCT, we design six variants from existing search algorithms: Genetic Algorithm, Particle Swarm Optimization and Ant Colony Algorithm by reversing and randomizing their mechanisms. We study their effectiveness in terms of generating a covering array and compare their performance. Experiments show that these search techniques can work well with distinct performance in covering array generation. We believe that these search techniques can be further improved by fine-tuning their configuration and used in broad ranges of area.

This paper introduces Model Projection. Finite state models such as Extended Finite State Machines are being used in an ever increasing number of software engineering activities. Model projection facilitates model development by specializing models for a specific operating environment. A projection is useful in many design-level applications including specification reuse and property verification.

Dependence analysis underpins many activities in software maintenance such as comprehension and impact analysis. As a result, dependence has been studied widely for programming languages, notably through work on program slicing. However, there is comparatively little work on dependence analysis at the model level and hitherto, no empirical studies. We introduce a slicing tool for Extended Finite State Machines (EFSMs) and use the tool to gather empirical results on several forms of dependence found in ten EFSMs, including well-known benchmarks in addition to real-world EFSM models. We investigate the statistical properties of dependence using statistical tests for correlation and formalize and prove four of the empirical findings arising from our empirical study. The paper thus provides the maintainer with both empirical data and foundational theoretical results concerning dependence in EFSM models.

This thesis presented a framework of the possible combination of approaches for low-level program slicing-based-dependence analysis and high-level concept assignment. Three combination techniques, concept extension, concept abbreviation and concept refinement, are presented and empirical studied to address the problem of program maintenance. The ten C subject programs are studied and more than 600 concept bindings are identified. Dependence based metrics are defined to evaluate three techniques that provides evidence of both advantages and disadvantages.

Key Statement Analysis extracts from a program, statements that form the core of the program’s computation. A good set of key statements is small but has a large impact. Key statements form a useful starting point for understanding and manipulating a program. An empirical investigation of three kinds of key statements is presented. The three are based on Bieman and Ott’s principal variables. To be effective, the key statements must have high impact and form a small, highly cohesive unit. Using a minor improvement of metrics for measuring impact and cohesion, key statements are shown to capture about 75% of the semantic effect of the function from which they are drawn. At the same time, they have cohesion about 20 percentage points higher than the corresponding function. A statistical analysis of the differences shows that key statements have higher average impact and higher average cohesion (p≪0.001).

The first ‘Testing: Academic & Industrial Conference—Practice and Research Techniques’ (TAIC PART 2006) was held at Cumberland Lodge Windsor during 29–31 August 2006. The general chair was Mark Harman (King's College London), the programme chair was Phil McMinn (University of Sheffield), and the local arrangements chair was Zheng Li (King's College London). TAIC PART is firmly grounded in fostering collaboration between industry and academia. It aims to bring together industrial software developers and users together with academic researchers working on the theory and practice of software testing. TAIC PART 2006 was a unique, not-for-profit conference that combined what we believe to be the best aspects of three kinds of event: a formal academic conference, a research workshop, and a retreat. The aim was to act not only as a forum for the exchange of ideas, but also as a vehicle to stimulate, deepen, and widen partnership between the academia and industry in software testing internationally. In all, 54 delegates from 11 different countries, comprising 32 academics and 22 industrialists, attended the conference. The event featured two keynotes, regular paper sessions, a PhD symposium, and a ‘speed dating’ session devoted to stimulating collaboration between attendees. The first keynote was given by Bill Woodworth, Corporate Director of IBM Quality Software Engineering. Bill spoke on test management at IBM, of which he has over 25 years of experience. John Hatcliff, who delivered the second keynote, is Professor in the Computing and Information Sciences Department at Kansas State University. John spoke on his internationally leading work on testing and software model checking. TAIC PART 2006 received a total of 50 full-paper submissions. After a rigorous reviewing process, 24 papers were accepted, with 8 of those papers from industry, 10 papers containing academic research, and a further 6 short papers accepted for the special PhD programme. Accepted papers covered a wide spectrum of state-of-the-art testing practice and research, including fault prediction, model-based testing, test specifications, the testing life cycle, search-based testing, database testing, web service testing, test requirements analysis, integration testing, empirical and case studies, and industrial challenges. The conference proceedings were published by IEEE and are available online. Two papers in this special issue are extended versions of some of the best papers originally presented at the conference. They have also been through a further reviewing process. The first paper is the result of an academic–industrial collaboration. In their paper ‘Quality Assurance for TTCN-3 Test Specifications’, Helmut Neukirchen, Benjamin Zeiss, and Jens Grabowski, of the University of Göttingen, and Paul Baker and Dominic Evans, of Motorola Labs, propose a technique to (1) assess the quality of existing test suites through metrics; (2) improve it through refactoring; and (3) detect refactoring opportunities by means of rules that are based on quality metrics. They focus on test suites expressed in the Testing and Test Control Notation TTCN-3, a language designed to support the specification of test suites in the telecommunication domain. The quality attribute considered in this work is maintainability, decomposed into analysability and changeability. Size, complexity, and coupling metrics are defined to characterize such quality attributes. The refactoring catalogue includes 23 TTCN-3-specific refactorings and 28 Java refactorings that are applicable to TTCN-3 as well. Eight rules are defined to check the applicability of refactorings automatically. These are implemented in a tool called ‘TRex’. The second paper, entitled ‘Automated Discovery of State Transitions and their Functions in Source Code’, by Neil Walkinshaw, Shaukat Ali, Kirill Bogdanov, and Mike Holcombe, presents a technique to reverse engineer source code into a state machine. It allows a developer to identify the states at a given point and statements that are responsible for state transitions. The technique also combines several ingredients, including symbolic execution and state abstraction, and is demonstrated with examples. Finally, we are grateful to our sponsors, whose financial contributions made it possible for TAIC PART 2006 to happen. Funding was received from the EPSRC and also from industry, including support from DaimlerChrysler, Ericsson, IPL Ltd., LDRA Ltd., Motorola, and Vizuri. The TAIC PART website (http://www2006.taicpart.org) serves as lasting resource to the event, containing the programme, photographs, and downloadable presentations of all the talks.

Regression testing is an expensive, but important, process. Unfortunately, there may be insufficient resources to allow for the reexecution of all test cases during regression testing. In this situation, test case prioritization techniques aim to improve the effectiveness of regression testing by ordering the test cases so that the most beneficial are executed first. Previous work on regression test case prioritization has focused on Greedy Algorithms. However, it is known that these algorithms may produce suboptimal results because they may construct results that denote only local minima within the search space. By contrast, metaheuristic and evolutionary search algorithms aim to avoid such problems. This paper presents results from an empirical study of the application of several greedy, metaheuristic, and evolutionary search algorithms to six programs, ranging from 374 to 11,148 lines of code for three choices of fitness metric. The paper addresses the problems of choice of fitness metric, characterization of landscape modality, and determination of the most suitable search technique to apply. The empirical results replicate previous results concerning Greedy Algorithms. They shed light on the nature of the regression testing search space, indicating that it is multimodal. The results also show that Genetic Algorithms perform well, although Greedy approaches are surprisingly effective, given the multimodal nature of the landscape.

One approach to supporting program comprehension involves binding concepts to source code. Previously proposed approaches to concept binding have enforced nonoverlapping boundaries. However, real-world programs may contain overlapping concepts. This paper presents techniques to allow boundary overlap in the binding of concepts to source code. In order to allow boundaries to overlap, the concept binding problem is reformulated as a search problem. It is shown that the search space of overlapping concept bindings is exponentially large, indicating the suitability of sampling-based search algorithms. Hill climbing and genetic algorithms are introduced for sampling the space. The paper reports on experiments that apply these algorithms to 21 COBOL II programs taken from the commercial financial services sector. The results show that the genetic algorithm produces significantly better solutions than both the hill climber and random search.