Generic controllers for self-adaptive systems can be configured parametrically according to system needs, even though their reuse is restricted because of the wide range of services that may be provided by each stage of a feedback control loop, like MAPE-K. Rainbow is a typical example of such a generic, monolithic controller. This article revisits and extends prior work that advocates structurally flexible controllers, as ensembles of micro-controllers each providing specific services. We experimented with our approach with three different architectural configurations for the controller—monolithic, decentralised and decentralised with a meta-controller. Our results indicate that despite the decentralised configuration with a meta-controller demanding more computational resources, it performed comparatively well compared to the other configurations, including when measuring the target system’s response time. Moreover, we found that variations of the control loop timing at the different layers of the controller impact the stability of the target system. We have also evolved the controller by adding a new micro-controller, which caused no impact on the other micro-controllers, and mostly kept the target system’s performance. We conclude that a multi-layered controller design, based on micro-controllers, provides the basis for defining structurally flexible controllers at operational-time and may promote reuse at development-time.

Mutation testing is a fault-based testing criterion that is used to measure the quality of the test suites of software systems. Due to its inherent high computational cost, many studies were published in the last decades aiming at reducing computational cost and human-effort for the mutation analysis. One of the most promising areas is searching for Strongly Subsuming Higher-Order Mutants (SSHOMs), which are rare and harder to kill than their constituent first-order mutants (FOMs). Therefore, they are valuable especially because they can replace their FOMs without loss of effectiveness in the mutation testing process. One of the explanations for the SSHOMs to be harder to kill than their constituent FOMs is that the single faults (mutations) can partially mask one another, so that the combination of them is harder to detect than any of the individual faults. However, we did not find in the literature an investigation of the masking phenomenon. Therefore, the goal of this paper is to start filling this gap. More specifically, for a mutation to mask the other one, it is necessary firstly that the execution of a test case reaches all involved mutations. Therefore, we designed two complementary studies to accomplish our goal. Study #1 focuses on searching for Strongly-Subsuming Second-Order Mutants (SS2OMs) and then analyzes reaching characteristics of their constituent FOMs. We found that almost half of the SS2OMs constituent FOMs are not even reaching the other FOM. In Study #2, we designed a search strategy that considers a second-order mutant killed only if both of their mutations are reached by a failing test case execution. This strategy found much more SS2OMs than in the first study.

In self-adaptive software systems, generic controllers can be configured parametrically according to system needs, even though their reuse is restricted because of the wide range of services that can be provided by each of the stages of a feedback control loop, like MAPE-K. Rainbow is a typical example of such a generic, monolithic controller. This paper advocates controllers that are structurally flexible, and which are composed from micro-controllers, each providing specific services (e.g., based on microservices). To provide evidence on the feasibility of our approach, we compare three different architectural configurations for the controller: monolithic, decentralised, and decentralised with a meta-controller. Results from our experiments indicate that even though the decentralised configuration with a meta-controller demanded more computational resources, it performed comparatively well when compared to the other configurations. We conclude that a multi-layered controller design, based on micro-controllers, provides the basis for defining structurally flexible controllers at operational-time, and may promote reuse at development-time.

Summary Model‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT:(i) some studies did not fully describe how tools transform abstract tests into concrete tests,(ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments.

Summary Adaptive systems (ASs) and context‐aware systems (CASs) are able to evaluate their own behaviour and toadaptit when the system fails to accomplish its goals or when better functionality or performance is possible. Ensuring the reliability of ASs and CASs is demanding because failures might have undesirable consequences. Testing ASs and CASs effectively is not trivial because of the inherent characteristics of these systems. The literature lacks a comprehensive review that provides a broad picture of the area; current reviews are outdated and incomplete. The objectives of this study are characterizing the state of the art in AS and CAS testing and discussing approaches, challenges, observed trends, and research limitations and directions. We performed a systematic literature review (SLR) and a thematic analysis of studies, reporting up‐to‐date, refined and extended results when compared with existing reviews. Based on 102 selected studies, we (i) characterized testing approaches by grouping techniques for ASs and CASs; (ii) updated and refined a characterization of testing challenges for ASs and CASs; and (iii) analysed and discussed research trends and implications for AS and CAS testing. There are recurring research concerns regarding AS and CAS testing. Examples are the generation of test cases and built‐in tests. Moreover, we also identified recurring testing challenges such as context monitoring and runtime decisions. Moreover, there are some trends such as model‐based testing and hybrid techniques and some little investigated issues like uncertainty and prediction of changes. All in all, our results may provide guidance for developers and researchers with respect to the practice and the future research on AS and CAS testing.

An adaptive system (AS) evaluates its own behavior and changes it when the evaluation indicates that the system is not accomplishing what it is intended to do, or when better functionality or performance is possible. MAPE-K is a reference model that prescribes the adaptation mechanism of ASs by means of high-level abstractions such as Monitors, Analyzers, Planners and Executors and the relationships among them. Since the abstractions and the relationships provided by MAPE-K are generic, other reference models were proposed focusing on providing lower level abstractions to support software engineers in a more suitable way. However, after the analysis of seven representative ASs, we realized the abstractions prescribed by the existing reference models are not properly implemented, thus leading to architectural drifts. Therefore, in this paper we characterized three of these drifts by describing them with a template and showing practical examples. The three architectural drifts of ASs are Scattered Reference Inputs, Mixed Executors and Effectors, and Obscure Alternatives. We expect that by identifying and characterizing these drifts, we can help software architects improve their design and, as a consequence, increase the reliability of this type of systems.

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

Software Testing (ST) is one of the least known aspects of software development. Yet, software engineers often argue that it demands more than half of the costs of a software project. Thus, proper testing education is of paramount importance. In fact, the mere exposition to ST knowledge might have an impact on programming skills. In particular, it can encourage the production of more reliable code. Although this is intuitive, to the best of our knowledge, there are no empirical studies about such effects. Evidence on this matter is important to motivate - or demotivate - classical testing education. Concerned with this, we have conducted a study to investigate the possible impact of ST knowledge on the production of reliable code. Our controlled experiment involved 28 senior-level Computer Science students, 8 auxiliary functions with 92 test cases, and a total of 112 implementations. Results show that code delivered after the exposition to ST knowledge is, on average, 20% more reliable (a significant difference at the 0.01 level). Also, implementations delivered afterwards are not significantly larger in terms of lines of code. This indicates that ST knowledge can make developers produce more reliable software with no additional overhead in terms of program size.

A considerable part of software systems is comprised of functions that support the main modules, such as array or string manipulation and basic math computation. These auxiliary functions are usually considered less complex, and thus tend to receive less attention from developers. However, failures in these functions might propagate to more critical modules, thereby affecting the system's overall reliability. Given the complementary role of auxiliary functions, a question that arises is whether agile practices, such as pair programming and test-first programming, can improve their correctness without affecting time-to-market. This paper presents an empirical assessment comparing the application of these agile practices with more traditional approaches. Our study comprises independent experiments of pair versus solo programming, and test-first versus test-last programming. The first study involved 85 novice programmers who applied both traditional and agile approaches in the development of six auxiliary functions within three different domains. Our results suggest that the agile practices might bring benefits in this context. In particular, pair programmers delivered correct implementations much more often, and test-first programming encouraged the production of larger and higher coverage test sets. On the downside, the main experiment showed that both practices significantly increase total development time. A replication of the test-first experiment with professional developers shows similar results.

Aspect-oriented programming (AOP) aims at facilitating program comprehension and maintenance in the presence of crosscutting concerns. Aspect code is often introduced and extended as the software projects evolve. Unfortunately, we still lack a good understanding of how faults are introduced in evolving aspect-oriented programs. More importantly, there is little knowledge whether existing metrics are related to typical fault introduction processes in evolving aspect-oriented code. This paper presents an exploratory study focused on the analysis of how faults are introduced during maintenance tasks involving aspects. The results indicate a recurring set of fault patterns in this context, which can better inform the design of future metrics for AOP. We also pinpoint AOP-specific fault categories which are difficult to detect with popular metrics for fault-proneness, such as coupling and code churn.

Due to the very large number of configurations that can typically be derived from a Dynamic Software Product Line (DSPL), efficient and effective testing of such systems have become a major challenge for software developers. In particular, when a configuration needs to be deployed quickly due to rapid contextual changes (e.g., in an unfolding crisis), time constraints hinder the proper testing of such a configuration. In this paper, we propose to reduce the testing required of such DSPLs to a relevant subset of configurations. Whenever a need to adapt to an untested configuration is encountered, our approach determines the most similar tested configuration and reuses its test results to either obtain a coverage measure or infer a confidence degree for the new, untested configuration. We focus on providing these techniques for inference of structural testing results for DSPLs, which is supported by an early prototype implementation.

This paper presents the results of an exploratory study on the fault-proneness of aspect-oriented programs. We analysed the faults collected from three evolving aspect-oriented systems, all from different application domains. The analysis develops from two different angles. Firstly, we measured the impact of the obliviousness property on the fault-proneness of the evaluated systems. The results show that 40% of reported faults were due to the lack of awareness among base code and aspects. The second analysis regarded the fault-proneness of the main aspect-oriented programming (AOP) mechanisms, namely pointcuts, advices and intertype declarations. The results indicate that these mechanisms present similar fault-proneness when we consider both the overall system and concern-specific implementations. Our findings are reinforced by means of statistical tests. In general, this result contradicts the common intuition stating that the use of pointcut languages is the main source of faults in AOP.

Coupling in software applications is often used as an indicator of external quality attributes such as fault-proneness. In fact, the correlation of coupling metrics and faults in object oriented programs has been widely studied. However, there is very limited knowledge about which coupling properties in aspect-oriented programming (AOP) are effective indicators of faults in modules. Existing coupling metrics do not take into account the specificities of AOP mechanisms. As a result, these metrics are unlikely to provide optimal predictions of pivotal quality attributes such as fault-proneness. This impacts further by restraining the assessments of AOP empirical studies. To address these issues, this paper presents an empirical study to evaluate the impact of coupling sourced from AOP-specific mechanisms. We utilise a novel set of coupling metrics to predict fault occurrences in aspect-oriented programs. We also compare these new metrics against previously proposed metrics for AOP. More specifically, we analyse faults from several releases of three AspectJ applications and perform statistical analyses to reveal the effectiveness of these metrics when predicting faults. Our study shows that a particular set of fine-grained directed coupling metrics have the potential to help create better fault prediction models for AO programs.

Software product lines (SPLs) enable modular, large-scale reuse through a software architecture addressing multiple core and varying features. To reap the benefits of SPLs, their designs need to be stable. Design stability encompasses the sustenance of the product line’s modularity properties in the presence of changes to both the core and varying features. It is usually assumed that aspect-oriented programming promotes better modularity and changeability of product lines. However, there is no empirical evidence on its efficacy to prolong design stability of product lines in realistic development scenarios. This paper reports a quantitative study that evolves two SPLs to assess various facets of design stability of aspect-oriented implementations. Our investigation focused upon a multi-perspective analysis of the evolving product lines in terms of modularity, change propagation, and feature interaction.

Mutation testing has been shown to be one of the strongest testing criteria for the evaluation of both programs and test suites. Comprehensive sets of mutants require strong test sets to achieve acceptable testing coverage. Moreover, mutation operators are valuable for the evaluation of other testing approaches. Although its importance has been highlighted for aspect-oriented (AO) programs, there is still a need for a suitable set of mutation operators for AO languages. The quality of the mutation testing itself relies on the quality of such operators. This paper presents the design of a set of mutation operators for AspectJ-based programs. These operators model instances of fault types identified in an extensive survey. The fault types and respective operators are grouped according to the related language features. We also discuss the generalisation of the fault types to AO approaches other than AspectJ and the coverage that may be achieved with the application of the proposed operators. In addition, a cost analysis based on two case studies involving real-world applications has provided us feedback on the most expensive operators, which will support the definition of further testing strategies.