Large Language Models (LLMs) are increasingly embedded in software via APIs like OpenAI, offering powerful AI features without heavy infrastructure. Yet these integrations bring their own form of self-admitted technical debt (SATD). In this paper, we present the first large-scale empirical study of LLM-specific SATD: its origins, prevalence, and mitigation strategies. By analyzing 93,142 Python files across major LLM APIs, we found that 54.49% of SATD instances stem from OpenAI integrations and 12.35% from LangChain use. Prompt design emerged as the primary source of LLM-specific SATD, with 6.61% of debt related to prompt configuration and optimization issues, followed by hyperparameter tuning and LLM-framework integration. We further explored which prompt techniques attract the most debt, revealing that instruction-based prompts (38.60%) and few-shot prompts (18.13%) are particularly vulnerable due to their dependence on instruction clarity and example quality. Finally, we release a comprehensive SATD dataset to support reproducibility and offer practical guidance for managing technical debt in LLM-powered systems.

Autonomous systems are susceptible to unknown safety issues due to overlooked dependencies among components of the system and the entities that are part of its operating environment. The current safety analysis techniques aids in identifying known safety issues but not overlooked/unknown safety issues. To identify unknown safety issues due to problematic interactions between components, in our previous work, we proposed safety assessment for concurrent components (SACC). Despite being more effective than FMEA and goal modeling, SACC suffers from some limitations such as not considering environmental entities and their properties, and a manual process for identifying associated components for the collective analysis. For a complex system with a large number of components, such an analysis can result in overlooking safety issues. To address these limitations, in this paper, we propose an association-driven safety analysis (ADSA) approach, which is extended and built on SACC. The approach uses a property-relation (PR) table and modified association rule mining algorithm to identify components and environmental entities that need to be considered together to detect overlooked or unknown safety issues. We evaluated our approach using four robotic systems and compared with SACC and systems theoretic process analysis (STPA). Our results show that our proposed approach, in particular using behavioral dependencies, is effective at exposing unknown safety issues.

Quality concerns are critical for addressing system-wide issues related to reliability, security, and performance, among others. However, these concerns often become scattered across the codebase, making it challenging for software developers to effectively address quality bugs. In this paper, we propose a holistic approach to detecting and clustering quality-related content hidden within the codebase. By leveraging the Hierarchical Dirichlet Process (HDP) and complementary techniques such as information retrieval and machine learning, including structural and textual analysis, we create a meaningful hierarchy that detects classes containing relevant information for addressing quality bugs. This approach allows us to uncover rich synergies between complex structured artifacts and infer bug-fixing classes for repairing quality bugs. The reported results show that our approach improves over the state-of-the-art achieving a high precision of 83%, recall of 82%, and F1 score of 83%.

Quality bugs are difficult to detect because the implemented quality-related features are commonly scattered across the codebase. Unfortunately, this scattered information prevents software developers from holistically understanding the root cause of quality bugs. The traditional view of a system does not support a hierarchical code view for monitoring and tracing how quality features are topically related and how they interact with each other. In this paper, we show how these limitations can be overcome by leveraging a Hierarchical Dirichlet Process (HDP) topic modeling technique along with other supporting intermediary techniques such as structural and textual analyses to capture hierarchical topical relationships among quality features across the codebase that yield to detection of quality bugs. We present SoftQualTopicDetector, that is capable of clustering scattered quality concerns into a meaningful hierarchy to infer a set of candidate classes relevant for recommending repair of quality bugs. The higher the ranking of classes into a hierarchy the more relevant they are regarded to contain information about the bug under investigation. Additionally, SoftQualTopicDetector incorporates three rich visualization features for monitoring, prioritizing, and 3-D tracing of suspicious classes to enhance aspects of maintainability, functional suitability, and tracability. We conduct an empirical evaluation of SoftQualTopicDetector that shows an improvement over the baseline and the state-of-the-art by terms ≈17% and in of average precision and ≈21% recall respectively.

Abstract In cyber‐physical systems such as robots and automated vehicles that rely heavily on batteries, safety, and energy conservation can result conflicting requirements when not considered together. In systems engineering, the development begins at a concept phase where we have high‐level information of different components of the system. During the concept phase, we perform hazard analysis and risk assessment, define safety goals, and derive safety requirements. This means requirement engineering occurs at the end of the concept phase. However, energy conservation recommendations are not taken into consideration until the detailed design with specific hardware and software is known. Hence, it is possible to recommend energy conservation behaviors that can compromise system's safety. If we perform a trade‐off analysis between safety and energy conservation at a concept phase, we can propose various design alternatives and choose the best one that offers a safe and energy saving architecture. To achieve this goal, we propose an approach for identifying safety issues that can be caused by energy conservation recommendations. To evaluate the effectiveness of our approach, we performed an empirical study on four robotic systems. Our results show that we can find energy conservation recommendations that can compromise safety at a concept phase.

In the early phases of a project, software architects and developers design solutions to satisfy quality concerns. However, as a byproduct of the long-term maintenance effort, qualities tend to erode, causing quality-related bugs to surface across the codebase. In principle, quality-related concerns not only can be expensive and difficult to detect, but they can have a detrimental effect on the system operating as intended. Moreover, quality-related concerns can directly affect users’ experiences at large. To address this problem, we build a quality-based bug classifier that leverages several feature selection techniques, TF-IDF, Chi-square (χ2), Mutual Information, and Extra Randomized Trees, including the incorporation of various machine learning algorithms. Our results indicate that Random Forest with the (TF-IDF+χ2) configuration achieved the best results for detecting six-quality related types, achieving a precision of 76%, recall of 70%, and F1 of 70%. However, the same approach returned low precision of 48%, recall of 15%, and F1 of 23% for detecting functional-related bugs. We argue that such low performance has resulted in an aftermath of overlapping content caused by functional and quality-related information which opens another challenging topic that we aim to expand in future work.

Mobile devices such as smart phones and smart watches utilize apps that run in context aware environments and must respond to context changes such as changes in network connectivity, battery level, screen orientation, and more.The large number of GUI events and context events often complicate the testing process.This work expands the AutoDroid tool to automatically generate tests that are guided by PairwiseInterleaved coverage of GUI event and context event sequences.We systematically weave context and GUI events into testing using the pairwise interleaved algorithm.The results show that the pairwise interleaved algorithm achieves up to five times higher code coverage compared to a technique that generates test suites in a single predefined context (without interleaving context and GUI events), a technique that changes the context at the beginning of each test case (without interleaving context and GUI events), and Monkey-Context-GUI (which randomly chooses context and GUI events).Future work will expand this strategy to include more context variables and test emerging technologies such as IoT and autonomous vehicles.

In recent years, there has been an increase in automation of safety critical systems such as self-driving cars, caretaking robots, or rescue drones. With increase in automation, the risk of systems behaving in an undesired manner has also risen. Most safety analysis approaches predominantly concentrate on identifying or assessing how the failure of a component can affect the behavior of a system based on the system's requirements. Yet, undesired behaviors can also occur when multiple components concurrently exert opposite effects on a shared resource. To identify safety-critical issues due to undesired concurrent component behaviors, we propose a property-driven approach called safety assessment for concurrent components (SACC). SACC uses a combinatorial technique that considers the requirements specification of a system, expressed as the states and properties of the system's components for identifying undesired combinations of component behaviors. To evaluate SACC, we performed a study using a requirements document on a caretaking robot. Our results show that SACC identified 38%-80% more undesired system behaviors when compared to the control techniques.

The use of risk information can help software engineers identify software components that are likely vulnerable or require extra attention when testing. Some studies have shown that the requirements risk-based approaches can be effective in improving the effectiveness of regression testing techniques. However, the risk estimation processes used in such approaches can be subjective, time-consuming, and costly. In this research, we introduce a fuzzy expert system that emulates human thinking to address the subjectivity related issues in the risk estimation process in a systematic and an efficient way and thus further improve the effectiveness of test case prioritization. Further, the required data for our approach was gathered by employing a semi-automated process that made the risk estimation process less subjective. The empirical results indicate that the new prioritization approach can improve the rate of fault detection over several existing test case prioritization techniques, while reducing threats to subjective risk estimation.

Off-nominal behaviors (ONBs) have been a major concern in the areas of embedded systems and safety-critical systems. To address ONB problems, some researchers have proposed model-based approaches that can expose ONBs by analyzing natural language requirements documents. While these approaches produced promising results, they require a lot of human effort and time. In this paper, to reduce human effort and time, we propose a combinatorial-based approach, Combinatorial Causal Component Model (Combi-CCM), which uses structured requirements patterns and combinations generated using the IPOG algorithm. We conducted an empirical study using several requirements documents to evaluate our approach, and our results indicate that the proposed approach can reduce human effort and time while maintaining the same ONB exposure ability obtained by the control techniques.

Regression test selection offers cost savings by selecting a subset of existing tests when testers validate the modified version of the application. The majority of test selection approaches utilize static or dynamic analyses to decide which test cases should be selected, and these analyses are often very time consuming. In this paper, we propose a novel language-independent Regression TEst SelecTion (ReTEST) technique that facilitates a lightweight analysis by using information retrieval. ReTEST uses fault history, test case diversity, and program change history information to select test cases that should be rerun. Our empirical evaluation with four open source programs shows that our approach can be effective and efficient by selecting a far smaller subset of tests compared to the existing techniques.

Finding software system defects during the requirements analysis phase can yield significant savings of time and effort when compared to finding the same defects during development or testing phases. The requirements engineering field has helped bring about significant advances in the early detection of system defects. However, a relatively small amount of research has been done regarding the detection of abnormal system behaviors. This is especially true for non-functional requirements (NFRs), which include areas such as timing and security requirements. Our work proposes the beginnings of a domain-specific modeling language for requirements analysis, with a particular emphasis on detecting abnormal system behaviors. We also demonstrate a preliminary version of our approach on a real-time embedded system.

Pervasive telemetry in modern applications is providing new possibilities in the application of regression testing techniques. Similar to how research in bioinformatics is leading to personalized medicine, tailored to individuals, usage telemetry in modern software allows for custom regression testing, tailored to the usage patterns of an installation. By customizing regression testing based on software usage, the effectiveness of regression testing techniques can be greatly improved, leading to reduced testing costs and enhanced detection of defects that are most important to that customer. In this research, we introduce the concept of fingerprinting software usage patterns through telemetry. We provide various algorithms tocompute fingerprints and conduct an empirical study that shows that fingerprints are effective in identifying distinct usage patterns. Further, we discuss how usage fingerprints can be used to improve regression test prioritization run time by over 30 percent compared to traditional prioritization techniques.

Software regression testing is an important, yet very costly, part of most major software projects. When regression tests run, any failures that are found help catch bugs early and smooth the future development work. The act of executing large numbers of tests takes significant resources that could, otherwise, be applied elsewhere. If tests could be accurately classified as likely to pass or fail prior to the run, it could save significant time while maintaining the benefits of early bug detection. In this paper, we present a case study to build a classifier for regression tests based on industrial software, Microsoft Dynamics AX. In this study, we examine the effectiveness of this classification as well as which aspects of the software are the most important in predicting regression test failures.

Regression testing is an important part of software development projects, and it is used to ensure software quality. Traditionally, a regression test focuses primarily on functional correctness of a modified program and is examined only when it fails, meaning it found a fault that would have otherwise been undetected. For certain application domains, regression tests for non-functional quality aspects such as performance, security, and usability could be just as important. However, those regression tests are much more costly and difficult to create, and thus many applications lack adequate non-functional regression test coverage. This adds risk of regressions in these areas as changes are made over time. In this research, we propose using metrics from passing test cases to predict quality aspects of the software beyond the traditional focus of regression tests. Our industrial case study shows that metrics such as test response time from functional regression tests are good predictors of which product areas are likely to contain certain types of non-functional performance faults. Furthermore, we show that this prediction can be improved through environmental perturbation such as the use of synthetic volume datasets or data size variation.

Requirements are typically specified on the assumption that the system's operating environment will behave in what is considered to be an expected and nominal manner. When gathering requirements, one concern is whether the requirements are too incomplete to account for every possible, unintended, off-nominal behavior (ONB) that the operating environment can create in the system. In this paper, we present a semi-automated approach, based on the causal component model (CCM), which can expose, within a set of requirements, whether ONBs can result in undesired system states. We demonstrate how the CCM approach exposes and helps address potential off-nominal behavior problems in a set of requirements that represents a real-world product. Our case study shows that the approach can expose susceptibility to ONBs and can supply information in correcting requirements.

Software regression testing is an integral part of most major software projects. As projects grow larger and the number of tests increases, performing regression testing becomes more costly. If software engineers can identify and run tests that are more likely to detect failures during regression testing, they may be able to better manage their regression testing activities. In this paper, to help identify such test cases, we developed techniques that utilizes various types of information in software repositories. To assess our techniques, we conducted an empirical study using an industrial software product, Microsoft Dynamics AX, which contains real faults. Our results show that the proposed techniques can be effective in identifying test cases that are likely to detect failures.

SUMMARY Companies that provide web applications often encounter various security attacks and frequent feature‐update demands from users, and when these needs arise, companies need to fix security problems or upgrade the application with new features. These fixes often involve small patches or revisions, but still, testers need to perform regression testing on their products to ensure that the changes have not introduced new faults. Performing regression testing on the entire product, however, can be very expensive, and it is not a viable solution for companies that need a short turnaround time to release patches. One solution is focusing only on the code areas that have been changed and performing regression testing on them. By doing this, companies can provide quick patches more dependably whenever they encounter security breaches. In this paper, the authors proposed a new regression testing approach that identifies the affected areas by code changes using impact analysis and generates new test cases for the impacted areas by changes using program slices. To facilitate the approach, the researchers implemented a Hypertext Preprocessor (PHP) Analysis and Regression Testing Engine (PARTE) and performed a controlled experiment using five open source web applications with multiple versions. The results showed that this approach is effective in reducing the cost of regression testing for a frequently patched web application, and exposed ways in which that effectiveness can vary with application characteristics and versioning frequencies. Copyright © 2014 John Wiley & Sons, Ltd.

Different testing environments and software change characteristics can affect the choice of regression testing techniques. In our prior work, we developed adaptive regression testing (ART) strategies to investigate this problem. While the ART strategies showed promising results, we also found that the multiple criteria decision making processes required for the ART strategies are time-consuming, often inaccurate and inconsistent, and limited in their scalability. To address these issues, in this research, we develop and empirically study a fuzzy expert system (FESART) to aid decision makers in choosing the most cost-effective technique for a particular software version. The results of our study show that FESART is consistently more cost-effective than the previously proposed ART strategies. One of the biggest contributors to FESART being more cost-effective is the reduced time required to apply the strategy. This contribution has significant impact because a strategy that is less time-consuming will be easier for researchers and practitioners to adopt, and will provide even greater cost-savings for regression testing sessions.

The importance of using requirements information in the testing phase has been well recognized by the requirements engineering community, but to date, a vast majority of regression testing techniques have primarily relied on software code information. Incorporating requirements information into the current testing practice could help software engineers identify the source of defects more easily, validate the product against requirements, and maintain software products in a holistic way. In this paper, we investigate whether the requirements-based clustering approach that incorporates traditional code analysis information can improve the effectiveness of test case prioritization techniques. To investigate the effectiveness of our approach, we performed an empirical study using two Java programs with multiple versions and requirements documents. Our results indicate that the use of requirements information during the test case prioritization process can be beneficial.

Testing game applications relies heavily on beta testing methods. The effectiveness of beta testing depends on how well beta testers represent the common game-application users and if users are willing to participate in the beta test. An automated testing tool framework could reduce the dependence upon beta testing by most companies to analyze their game applications. This paper presents the Crushinator as one such framework. This framework provides a game-independent testing tool that implements multiple testing methods that can assist and possibly replace the use of beta testing.

SUMMARY Software security issues have been a major concern in the cyberspace community, so a great deal of research on security testing has been performed, and various security testing techniques have been developed. Threat modeling provides a systematic way to identify threats that might compromise security, and it has been a well‐accepted practice by the industry, but test case generation from threat models has not been addressed yet. Thus, in this paper, we propose a threat model‐based security testing approach that automatically generates security test sequences from threat trees and transforms them into executable tests. The security testing approach we consider consists of three activities in large: building threat models with threat trees; generating security test sequences from threat trees; and creating executable test cases by considering valid and invalid inputs. To support our approach, we implemented security test generation techniques, and we also conducted an empirical study to assess the effectiveness of our approach. The results of our study show that our threat tree‐based approach is effective in exposing vulnerabilities. Copyright © 2012 John Wiley & Sons, Ltd.

Web applications change and are upgraded frequently due to security attacks, feature updates, or user preference changes. These fixes often involve small patches or revisions, but still, testers need to perform regression testing on their products to ensure that the changes have not introduced new faults. Applying regression testing to the entire product, however, can be very expensive, and often, companies cannot afford to do this because, typically, the turnaround time to release patches is expected to be short. One solution is focusing only on the areas of code that have been changed and performing regression testing on them. In this way, companies can provide quick patches more dependably whenever they encounter security breaches. In this paper, we propose a new regression testing approach that is applied to frequently patched web applications, considering security problems, and in particular, focusing on PHP programs. Our approach identifies the affected areas by code changes using impact analysis and generates new test cases for the impacted areas by changes using program slices considering both numeric and string input values. To facilitate our approach, we implemented a PHP Analysis and Regression Testing Engine (PARTE) and performed a controlled experiment using open source web applications. The results show that our approach is effective in reducing the cost of regression testing for frequently patched web applications.

Regression testing is an important activity for controlling the quality of a software product, but it accounts for a large proportion of the costs of software. We believe that an understanding of the underlying relationships in data about software systems, including data correlations and patterns, could provide information that would help improve regression testing techniques. We conjecture that if test cases have common properties, then test cases within the same group may have similar fault detection ability. As an initial approach to investigating the relationships in massive data in software repositories, in this paper, we consider a clustering approach to help improve test case prioritization. We implemented new prioritization techniques that incorporate a clustering approach and utilize code coverage, code complexity, and history data on real faults. To assess our approach, we have designed and conducted empirical studies using an industrial software product, Microsoft Dynamics Ax, which contains real faults. Our results show that test case prioritization that utilizes a clustering approach can improve the effectiveness of test case prioritization techniques.

When software systems evolve, different amounts and types of code modifications can be involved in different versions. These factors can affect the costs and benefits of regression testing techniques in different ways, and thus, there may be no single regression testing technique that is the most cost-effective technique to use on every version. To date, many regression testing techniques have been proposed, but no research has been done on the problem of helping practitioners systematically choose appropriate techniques on new versions as systems evolve. To address this problem, we propose adaptive regression testing (ART) strategies that attempt to identify the regression testing techniques that will be the most cost-effective for each regression testing session considering organization's situations and testing environment. To assess our approach, we conducted an experiment focusing on test case prioritization techniques. Our results show that prioritization techniques selected by our approach can be more cost-effective than those used by the control approaches.

Adequate requirements validation could prevent errors from propagating into later development phase, and eventually improve the quality of software systems. However, often validating textual requirements is difficult and error prone. We develop a feedback-based requirements validation methodology that provides an interactive and systematic way to validate a requirements model. Our approach is based on the notion of querying a model, which is built from a requirements specification, with scenario questions, in order to determine whether the model's behavior satisfies the given requirements. To investigate feasibility of our approach, we implemented a Scenario Question Query Engine (SQ2E), which uses scenario questions to query a model, and performed a preliminary case study using a real-world application. The results show that the approach we proposed was effective in detecting both expected and unexpected behaviors in a model. We believe that our approach could improve the quality of requirements and ultimately the quality of software systems.

Regression testing is a necessary maintenance activity that can ensure high quality of the modified software system, and a great deal of research on regression testing has been performed. Most of the studies performed to date, however, have evaluated regression testing techniques under the limited context, such as a short-term assessment, which do not fully account for system evolution or industrial circumstances. One important issue associated with a system lifetime view that we have overlooked in past years is the effects of residual defects - defects that persist undetected - across several releases of a system. Depending on an organization's business goals and the type of system being built, residual defects might affect the level of success of the software products. In this paper, we conducted an empirical study to investigate whether regression testing techniques are effective in reducing the occurrence and persistence of residual defects across a system's lifetime, in particular, considering test case prioritization techniques. Our results show that heuristics can be effective in reducing both the occurrence of residual defects and their age. Our results also indicate that residual defects and their age have a strong impact on the cost-benefits of test case prioritization techniques.

Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. If this is correct, a better understanding of the effects of time constraints could lead to improved prioritization techniques and improved maintenance and testing processes. We therefore conducted a series of experiments to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our first experiment manipulates time constraint levels and shows that time constraints do play a significant role in determining both the cost-effectiveness of prioritization and the relative cost-benefit trade-offs among techniques. Our second experiment replicates the first experiment, controlling for several threats to validity including numbers of faults present, and shows that the results generalize to this wider context. Our third experiment manipulates the number of faults present in programs to examine the effects of faultiness levels on prioritization and shows that faultiness level affects the relative cost-effectiveness of prioritization techniques. Taken together, these results have several implications for test engineers wishing to cost-effectively regression test their software systems. These include suggestions about when and when not to prioritize, what techniques to employ, and how differences in testing processes may relate to prioritization cost--effectiveness.

Software engineering methodologies are subject to complex cost-benefit tradeoffs. Economic models can help practitioners and researchers assess methodologies relative to these tradeoffs. Effective economic models, however, can be established only through an iterative process of refinement involving analytical and empirical methods. Sensitivity analysis provides one such method. By identifying the factors that are most important to models, sensitivity analysis can help simplify those models; it can also identify factors that must be measured with care, leading to guidelines for better test strategy definition and application. In prior work we presented the first comprehensive economic model for the regression testing process, that captures both cost and benefit factors relevant to that process while supporting evaluation of these processes across entire system lifetimes. In this work we use sensitivity analysis to examine our model analytically and assess the factors that are most important to the model. Based on the results of that analysis, we propose two new models of increasing simplicity. We assess these models empirically on data obtained by using regression testing techniques on several non-trivial software systems. Our results show that one of the simplified models assesses the relationships between techniques in the same way as the full model.

Regression testing is an expensive process used to validate modified software. Test case prioritization techniques improve the cost-effectiveness of regression testing by ordering test cases such that those that are more important are run earlier in the testing process. Many prioritization techniques have been proposed and evidence shows that they can be beneficial. It has been suggested, however, that the time constraints that can be imposed on regression testing by various software development processes can strongly affect the behavior of prioritization techniques. Therefore, we conducted an experiment to assess the effects of time constraints on the costs and benefits of prioritization techniques. Our results show that time constraints can indeed play a significant role in determining both the cost-effectiveness of prioritization, and the relative cost-benefit tradeoffs among techniques, with important implications for the use of prioritization in practice.

Abstract Increasingly, modern‐day software systems are being built by combining externally‐developed software components with application‐specific code. For such systems, existing program‐analysis‐based software engineering techniques may not directly apply, due to lack of information about components. To address this problem, the use of component metadata has been proposed. Component metadata are metadata and metamethods provided with components, that retrieve or calculate information about those components. In particular, two component‐metadata‐based approaches for regression test selection are described: one using code‐based component metadata and the other using specification‐based component metadata. The results of empirical studies that illustrate the potential of these techniques to provide savings in re‐testing effort are provided. Copyright © 2006 John Wiley & Sons, Ltd.

Regression testing is an important but expensive activity, and a great deal of research on regression testing methodologies has been performed. In recent years, much of this research has emphasized empirical studies, including evaluations of the effectiveness and efficiency of regression testing techniques. To date, however, most studies have been limited in terms of their consideration of testing context and system lifetime, and have used cost-benefit models that omit important factors and render some types of comparisons between techniques impossible. These limitations can cause studies to improperly assess the costs and benefits of regression testing techniques in practical settings. In this paper, we provide improved cost-benefit models for use in assessing regression testing methodologies, that incorporate context and lifetime factors not considered in prior studies, and we use these models to compare several common methodologies. Our results show that the factors we consider (in particular, time constraints and incremental resource availability) can affect assessments of the relative benefits of regression testing techniques, and suggest that particular classes of techniques may compare differently across different types of test suites.

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general.

Regression testing is an important part of software maintenance, but it can also be very expensive. To reduce this expense, software testers may prioritize their test cases so that those that are more important are run earlier in the regression testing process. Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited to hand-seeded faults, primarily due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults. We have therefore designed and performed a controlled experiment to assess the ability of prioritization techniques to improve the rate of fault detection techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. We also compare our results to those collected earlier with respect to the relationship between hand-seeded faults and mutation faults, and the implications this has for researchers performing empirical studies of prioritization.

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites, in particular, Java and the JUnit testing framework are being used extensively in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm.

Component based software technologies are viewed as essential for creating the software systems of the future. However, the use of externally-provided components has serious drawbacks for a wide range of software engineering activities, often because of a lack of information about the components. Previously (A. Orso et al., 2000), we proposed the use of component metacontents: additional data and methods provided with a component, to support software engineering tasks. The authors present two new metacontent based techniques that address the problem of regression test selection for component based applications: a code based approach and a specification based approach. First, we illustrate the two techniques. Then, we present a case study that applies the code based technique to a real component based system. On the system studied, on average, 26% of the overall testing effort was saved over seven releases, with a maximum savings of 99% for one version.