Abstract Assertions are one of the most useful automated techniques for checking program’s behaviour and hence have been used for different verification and validation tasks. We provide an overview of the last two decades of research involving “assertions” in software testing. Based on a term-based search, we filtered the inclusion of relevant papers and synthesised them with respect to the problem addressed, the solution designed, and the evaluation conducted. The survey rendered 145 papers on assertions in software testing. After test oracle, the dominant problem is test generation, followed by engineering aspects of assertions. Solutions are typically embedded in tool prototypes and evaluated throughout a limited number of cases, whereas using large-scale industrial settings is still a noticeable method. We conclude that assertions would be worth more attention in future research, particularly regarding the new and emerging demands (e.g., verification of programs with uncertainty), for effective, applicable, and domain-specific solutions.

Regression testing is a means to assure that a change in the software, or its execution environment, does not introduce new defects. It involves the expensive undertaking of rerunning test cases. Several techniques have been proposed to reduce the number of test cases to execute in regression testing, however, there is no research on how to assess industrial relevance and applicability of such techniques. We conducted a systematic literature review with the following two goals: firstly, to enable researchers to design and present regression testing research with a focus on industrial relevance and applicability and secondly, to facilitate the industrial adoption of such research by addressing the attributes of concern from the practitioners’ perspective. Using a reference-based search approach, we identified 1068 papers on regression testing. We then reduced the scope to only include papers with explicit discussions about relevance and applicability (i.e. mainly studies involving industrial stakeholders). Uniquely in this literature review, practitioners were consulted at several steps to increase the likelihood of achieving our aim of identifying factors important for relevance and applicability. We have summarised the results of these consultations and an analysis of the literature in three taxonomies, which capture aspects of industrial-relevance regarding the regression testing techniques. Based on these taxonomies, we mapped 38 papers reporting the evaluation of 26 regression testing techniques in industrial settings.

This study presents a process‐driven view on the use of model‐based testing (MBT) in agile/lightweight processes. It argues that process‐related aspects of MBT and agile processes should be explicitly considered in any practical MBT approach intended for use in agile development (AD). It demonstrates that an effective MBT approach for lightweight processes has to specify how MBT activities are integrated into a development process, how and when MBT artefacts are generated in relation to other development artefacts, and who would carry out MBT activities. Accordingly, a set of integration criteria is introduced for complete incorporation of MBT into agile/lightweight processes. The integration criteria demonstrate the specific characteristics of an appropriate MBT process for AD processes, and help identify the benefits and shortcomings of existing methods on the use of MBT in such processes. Evaluation of existing works based on the proposed integration criteria shows that they have all focused on minimal modelling , and only one method has considered the ‘evolution’ of test models and the ‘reuse’ of test cases, whereas ‘evolution’ and ‘reuse’ are essential characteristics of agile processes, which have to be addressed in any MBT approach intended for such processes.

Agile software development is regarded as an effective and efficient approach, mainly due to its ability to accommodate rapidly changing requirements, and to cope with modern software development challenges. There is therefore a strong tendency to use agile software development methodologies where applicable; however, the sheer number of existing agile methodologies and their variants hinders the selection of an appropriate agile methodology or method chunk. Methodology evaluation tools address this problem through providing detailed evaluations, yet no comprehensive evaluation framework is available for agile methodologies. We introduce the comprehensive evaluation framework for agile methodologies (CEFAM) as an evaluation tool for project managers and method engineers. The hierarchical (and mostly quantitative) evaluation criterion set introduced in this evaluation framework enhances the usability of the framework and provides results that are precise enough to be useful for the selection, adaptation and construction of agile methodologies.