Peer evaluation is widely used in team project courses in software engineering to assess individual contributions, promote accountability, and improve teamwork. Existing peer evaluation practices vary greatly in their purpose, design, and implementation, making it difficult to compare them and to establish best practices. This study systematically investigates peer evaluation in team project courses in software engineering. It introduces a taxonomy to categorize peer evaluation practices based on their purpose, participants, mechanisms, and outcomes. Using this taxonomy, we organize findings from existing literature and characterize how peer evaluation is applied across different courses. We further examine educators’ perspectives on the quality and applicability of the taxonomy. Based on the educators' feedback, we updated the taxonomy and provided a set of guidelines to support educators and researchers in designing and implementing peer evaluation interventions and reporting them in a structured and informed way.

[Context] The lack of practical relevance in many Software Engineering (SE) research contributions is often rooted in oversimplified views of industrial practice, weak industry connections, and poorly defined research problems. Clear criteria for evaluating SE research problems can help align their value, feasibility, and applicability with industrial needs. [Goal] In this paper, we introduce the Lean Research Inception (LRI) framework, designed to support the formulation and assessment of practically relevant research problems in SE. We describe its initial evaluation strategy conducted in a workshop with a network of SE researchers experienced in industry-academia collaboration and report the evaluation of its three assessment criteria (valuable, feasible, and applicable) regarding their importance and completeness in assessing practical relevance. [Method] We applied LRI retroactively to a published research paper, engaging workshop participants in discussing and assessing the research problem by applying the proposed criteria using a semantic differential scale. Participants provided feedback on the criteria’s importance and completeness, drawn from their own experiences in industry-academia collaboration. [Results] The findings reveal an overall agreement on the importance of the three criteria – valuable (83.3%), feasible (76.2%), and applicable (73.8%) – for aligning research problems with industrial needs. Qualitative feedback suggested adjustments in terminology with a clearer distinction between feasible and applicable, and refinements for valuable by more clearly considering business value, ROI, and originality. [Conclusion] While LRI still constitutes ongoing research and requires further evaluation, our emerging results strengthen our confidence that the three criteria applied using the semantic differential scale can already help the community assess the practical relevance of SE research problems.

Several studies have evaluated various ML techniques and found promising results in classifying bug reports . However, these studies have used different evaluation designs, making it difficult to compare their results. Furthermore, they have focused primarily on accuracy and did not consider other potentially relevant factors such as generalizability , explainability, and maintenance cost. These two aspects make it difficult for practitioners and researchers to choose an appropriate ML technique for a given context. Therefore, we compare promising ML techniques against practitioners’ concerns using evaluation criteria that go beyond accuracy. Based on an existing framework for adopting ML techniques, we developed an evaluation framework for ML techniques for bug report classification. We used this framework to compare nine ML techniques on three datasets. The results enable a tradeoff analysis between various promising ML techniques. The results show that an ML technique with the highest predictive accuracy might not be the most suitable technique for some contexts. The overall approach presented in the paper supports making informed decisions when choosing ML techniques. It is not locked to the specific techniques, datasets, or factors we have selected here, and others could easily use and adapt it for additional techniques or concerns. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Team project courses in software engineering allow students to apply their acquired disciplinary knowledge while developing essential skills needed to work in the software industry. This paper examines the challenges and lessons learned by students in two team project courses involving industrial customers. The first course involves small teams and less complex project, whereas the second course, has larger teams and more complex projects. Using thematic analysis, we analyzed 158 reports submitted by two cohorts of students across two successive team project courses. As per our findings most challenges and lessons learned pertain to soft skills, such as teamwork, working in remote and hybrid setting, and collaboration with industrial customers. The results show that challenges and lessons learned evolve as students progress to the second team project course, for example, managing changes and addressing individual skill gaps were more pronounced in the first project course, while students reported greater coordination, communication, and contribution issues in the second team project course. The alignment between the challenges faced and the lessons learned suggests that addressing challenges in teamwork, collaborating with industrial customers, and working in hybrid or remote settings helped students develop effective strategies to mitigate these challenges. This process offers a valuable learning experience for the students, enriching their professional growth. • We analyze students’ reports from two cohorts across two team project courses. • We analyze the evolution of challenges and lessons learned across two courses. • Most challenges and lessons learned relate to soft skills. • Involving industrial customers in team project courses enhances student learning. • Multiple team project courses improve students’ learning experiences.

The evidence for the prevalence of test smells at the unit testing level has relied on the accuracy of detection tools, which have seen intense research in the last two decades. The Eager Test smell, one of the most prevalent, is often identified using simplified detection rules that practitioners find inadequate. We aim to improve the rules for detecting the Eager Test smell. We reviewed the literature on test smells to analyze the definitions and detection rules of the Eager Test smell. We proposed a novel, unambiguous definition of the test smell and a heuristic to address the limitations of the existing rules. We evaluated our heuristic against existing detection rules by manually applying it to 300 unit test cases in Java. Our review identified 56 relevant studies. We found that inadequate interpretations of original definitions of the Eager Test smell led to imprecise detection rules, resulting in a high level of disagreement in detection outcomes. Also, our heuristic detected patterns of eager and non-eager tests that existing rules missed. Our heuristic captures the essence of the Eager Test smell more precisely; hence, it may address practitioners’ concerns regarding the adequacy of existing detection rules. • An overview of the definitions and detection rules of the Eager Test smell. • A novel, unambiguous definition of Eager Test based on the original definitions. • A novel heuristic with pseudocode for detecting eager tests. • The identification of common patterns of eager tests and non-eager tests.

Abstract Context: Code reviewers provide valuable feedback during the code review. Identifying common issues described in the reviewers’ feedback can provide input for devising context-specific software development improvements. However, the use of reviewer feedback for this purpose is currently less explored. Objective: In this study, we assess how automation can derive more interpretable and informative themes in reviewers’ feedback and whether these themes help to identify recurring quality-related issues in code changes. Method: We conducted a participatory case study using the JabRef system to analyze reviewers’ feedback on merged and abandoned code changes. We used two promising topic modeling methods (GSDMM and BERTopic) to identify themes in 5,560 code review comments. The resulting themes were analyzed and named by a domain expert from JabRef. Results: The domain expert considered the identified themes from the two topic models to represent quality-related issues. Different quality issues are pointed out in code reviews for merged and abandoned code changes. While BERTopic provides higher objective coherence, the domain expert considered themes from short-text topic modeling more informative and easy to interpret than BERTopic-based topic modeling. Conclusions: The identified prevalent code quality issues aim to address the maintainability-focused issues. The analysis of code review comments can enhance the current practices for JabRef by improving the guidelines for new developers and focusing discussions in the developer forums. The topic model choice impacts the interpretability of the generated themes, and a higher coherence (based on objective measures) of generated topics did not lead to improved interpretability by a domain expert.

Abstract The quality of the test suites and the constituent test cases significantly impacts confidence in software testing. While research has identified several quality attributes of test cases and test suites, there is a need for a better understanding of their relative importance in practice. We investigate practitioners’ perceptions regarding the relative importance of quality attributes of test cases and test suites and the challenges that they face in ensuring the perceived important quality attributes. To capture the practitioners’ perceptions, we conducted an industrial survey using a questionnaire based on the quality attributes identified in an extensive literature review. We used a sampling strategy that leverages LinkedIn to draw a large and heterogeneous sample of professionals with experience in software testing. We collected 354 responses from practitioners with a wide range of experience (from less than one year to 42 years of experience). We found that the majority of practitioners rated Fault Detection, Usability, Maintainability, Reliability, and Coverage to be the most important quality attributes. Resource Efficiency, Reusability, and Simplicity received the most divergent opinions, which, according to our analysis, depend on the software-testing contexts. Also, we identified common challenges that apply to the important attributes, namely inadequate definition, lack of useful metrics, lack of an established review process, and lack of external support. The findings point out where practitioners actually need further support with respect to achieving high-quality test cases and test suites under different software testing contexts. Hence, the findings can serve as a guideline for academic researchers when looking for research directions on the topic. Furthermore, the findings can be used to encourage companies to provide more support to practitioners to achieve high-quality test cases and test suites.

Characterising code quality is a challenge that was addressed by Börstler et al. 's working group in 2017. As emerged from their study, educators, developers and students have different perceptions of the manifold aspects involved, and a major conclusion of that WG was that "code quality should be discussed more thoroughly in educational programs" [2, p. 70]. However, the lack of materials and the time constraints have slowed down progress in that regard.

Abstract Despite the accuracy of machine learning (ML) techniques in predicting invalid bug reports, as shown in earlier research, and the importance of early identification of invalid bug reports in software maintenance, the adoption of ML techniques for this task in industrial practice is yet to be investigated. In this study, we used a technology transfer model to guide the adoption of an ML technique at a company for the early identification of invalid bug reports. In the process, we also identify necessary conditions for adopting such techniques in practice. We followed a case study research approach with various design and analysis iterations for technology transfer activities. We collected data from bug repositories, through focus groups, a questionnaire, and a presentation and feedback session with an expert. As expected, we found that an ML technique can identify invalid bug reports with acceptable accuracy at an early stage. However, the technique’s accuracy drops over time in its operational use due to changes in the product, the used technologies, or the development organization. Such changes may require retraining the ML model. During validation, practitioners highlighted the need to understand the ML technique’s predictions to trust the predictions. We found that a visual (using a state-of-the-art ML interpretation framework) and descriptive explanation of the prediction increases the trustability of the technique compared to just presenting the results of the validity predictions. We conclude that trustability, integration with the existing toolchain, and maintaining the techniques’ accuracy over time are critical for increasing the likelihood of adoption.

The adoption or acceptance of new technologies or ways of working in software development activities is a recurrent topic in the software engineering literature. The topic has, therefore, been empirically investigated extensively. It is, however, unclear which theoretical frames of reference are used in this research to explain acceptance behaviors. In this study, we explore how major theories and models of acceptance behavior have been used in the software engineering literature to empirically investigate acceptance behavior. We conduct a systematic mapping study of empirical studies using acceptance behavior theories in software engineering. We identified 47 primary studies covering 56 theory uses. The theories were categorized into six groups. Technology acceptance models (TAM and its extensions) were used in 29 of the 47 primary studies, innovation theories in 10, and the theories of planned behavior/ reasoned action (TPB/TRA) in six. All other theories were used in at most two of the primary studies. The usage and operationalization of the theories were, in many cases, inconsistent with the underlying theories. Furthermore, we identified 77 constructs used by these studies of which many lack clear definitions. Our results show that software engineering researchers are aware of some of the leading theories and models of acceptance behavior, which indicates an attempt to have more theoretical foundations. However, we identified issues related to theory usage that make it difficult to aggregate and synthesize results across studies. We propose mitigation actions that encourage the consistent use of theories and emphasize the measurement of key constructs.

Several secondary studies have investigated the relationship between internal quality attributes, source code metrics and external quality attributes. Sometimes they have contradictory results. We synthesize evidence of the link between internal quality attributes, source code metrics and external quality attributes along with the efficacy of the prediction models used. We conducted a tertiary review to identify, evaluate and synthesize secondary studies. We used several characteristics of secondary studies as indicators for the strength of evidence and considered them when synthesizing the results. From 711 secondary studies, we identified 15 secondary studies that have investigated the link between source code and external quality. Our results show : (1) primarily, the focus has been on object-oriented systems, (2) maintainability and reliability are most often linked to internal quality attributes and source code metrics, with only one secondary study reporting evidence for security, (3) only a small set of complexity, coupling, and size-related source code metrics report a consistent positive link with maintainability and reliability, and (4) group method of data handling (GMDH) based prediction models have performed better than other prediction models for maintainability prediction. Based on our results, lines of code, coupling, complexity and the cohesion metrics from Chidamber & Kemerer (CK) metrics are good indicators of maintainability with consistent evidence from high and moderate-quality secondary studies. Similarly, four CK metrics related to coupling, complexity and cohesion are good indicators of reliability, while inheritance and certain cohesion metrics show no consistent evidence of links to maintainability and reliability. Further empirical studies are needed to explore the link between internal quality attributes, source code metrics and other external quality attributes, including functionality, portability, and usability. The results will help researchers and practitioners understand the body of knowledge on the subject and identify future research directions.

Context: The modern code review process is considered an essential quality assurance step in software development. The code review comments generated can provide insights regarding source code quality and development practices. However, the large number of code review comments makes it challenging to identify interesting patterns manually. In a recent study, Wen et al. used traditional topic modeling to analyze the evolution of code review comments. Their approach could identify interesting patterns that may lead to improved development practices.Objective: In this study, we investigate potential improvements to Wen et al.’s state-of-the-art approach to analyze the evolution of code review comments.Method: We used 209,166 code review comments from three open-source systems to explore and empirically analyze alternative design and implementation choices and demonstrate their impact.Results: We identified the following potential improvements to the current state-of-the-art as described by Wen et al.: 1) utilize a topic modeling method that is optimized for short texts, 2) a refined approach for identifying a suitable number of topics, and 3) a more elaborate approach for analyzing topic evolution. Our results indicate that the proposed changes have quantitatively different results than the current approach. The qualitative interpretation of the topics generated with our changes indicates their usefulness.Conclusions: Our results indicate the potential usefulness of changes to state-of-the-art approaches to analyzing the evolution of code review comments, with practical implications for researchers and practitioners. However, further research is required to compare the effectiveness of both approaches.

Abstract There are many aspects of code quality, some of which are difficult to capture or to measure. Despite the importance of software quality, there is a lack of commonly accepted measures or indicators for code quality that can be linked to quality attributes. We investigate software developers’ perceptions of source code quality and the practices they recommend to achieve these qualities. We analyze data from semi-structured interviews with 34 professional software developers, programming teachers and students from Europe and the U.S. For the interviews, participants were asked to bring code examples to exemplify what they consider good and bad code, respectively. Readability and structure were used most commonly as defining properties for quality code. Together with documentation, they were also suggested as the most common target properties for quality improvement. When discussing actual code, developers focused on structure, comprehensibility and readability as quality properties. When analyzing relationships between properties, the most commonly talked about target property was comprehensibility. Documentation, structure and readability were named most frequently as source properties to achieve good comprehensibility. Some of the most important source code properties contributing to code quality as perceived by developers lack clear definitions and are difficult to capture. More research is therefore necessary to measure the structure, comprehensibility and readability of code in ways that matter for developers and to relate these measures of code structure, comprehensibility and readability to common software quality attributes.

Double-counting in a literature review occurs when the same data, population, or evidence is erroneously counted multiple times during synthesis. Detecting and mitigating the threat of double-counting is particularly challenging in tertiary studies. Although this topic has received much attention in the health sciences, it seems to have been overlooked in software engineering. We describe issues with double-counting in tertiary studies, investigate the prevalence of the issue in software engineering, and propose ways to identify and address the issue. We analyze 47 tertiary studies in software engineering to investigate in which ways they address double-counting and whether double-counting might be a threat to validity in them. In 19 of the 47 tertiary studies, double-counting might bias their results. Of those 19 tertiary studies, only 5 consider double-counting a threat to their validity, and 7 suggest strategies to address the issue. Overall, only 9 of the 47 tertiary studies, acknowledge double-counting as a potential general threat to validity for tertiary studies. Double-counting is an overlooked issue in tertiary studies in software engineering, and existing design and evaluation guidelines do not address it sufficiently. Therefore, we propose recommendations that may help to identify and mitigate double-counting in tertiary studies.

Bug reports created during software development and maintenance do not always describe deviations from a system’s valid behavior. Such invalid bug reports may consume significant resources and adversely affect the prioritization and resolution of valid bug reports. There is a need to identify preventive actions to reduce the inflow of invalid bug reports. Existing research has shown that manually analyzing invalid bug report descriptions provides cues regarding preventive actions. However, such a manual approach is not cost-effective due to the time required to analyze a sufficiently large number of bug reports needed to identify useful patterns. Furthermore, the analysis needs to be repeated as the underlying causes of invalid bug reports change over time. In this study, we propose and evaluate the use of Latent Dirichlet Allocation (LDA), a topic modeling approach, to support practitioners in suggesting preventive actions to avoid the creation of similar invalid bug reports in the future. In an industrial case study, we first manually analyzed descriptions of invalid bug reports to identify common patterns in their descriptions. We further investigated to what extent LDA can support this manual process. We used expert-based validation to evaluate the relevance of identified common patterns and their usefulness in suggesting preventive measures. We found that invalid bug reports have common patterns that are perceived as relevant, and they can be used to devise preventive measures. Furthermore, the identification of common patterns can be supported with automation. Using LDA, practitioners can effectively identify representative groups of bug reports (i.e., relevant common patterns) from a large number of bug reports and analyze them further to devise preventive measures.

Software engineering research aims to establish software development practice on a scientific basis. However, the evidence of the efficacy of technology is insufficient to ensure its uptake in industry. In the absence of a theoretical frame of reference, we mainly rely on best practices and expert judgment from industry-academia collaboration and software process improvement research to improve the acceptance of the proposed technology. To identify acceptance models and theories and discuss their applicability in the research of acceptance behavior related to software development. We analyzed literature reviews within an interdisciplinary team to identify models and theories relevant to software engineering research. We further discuss acceptance behavior from the human information processing perspective of automatic and affect-driven processes (“fast” system 1 thinking) and rational and rule-governed processes (“slow” system 2 thinking). We identified 30 potentially relevant models and theories. Several of them have been used in researching acceptance behavior in contexts related to software development, but few have been validated in such contexts. They use constructs that capture aspects of (automatic) system 1 and (rational) system 2 oriented processes. However, their operationalizations focus on system 2 oriented processes indicating a rational view of behavior, thus overlooking important psychological processes underpinning behavior. Software engineering research may use acceptance behavior models and theories more extensively to understand and predict practice adoption in the industry. Such theoretical foundations will help improve the impact of software engineering research. However, more consideration should be given to their validation, overlap, construct operationalization, and employed data collection mechanisms when using these models and theories.

Practitioners working in large-scale software development face many challenges in regression testing activities. One of the reasons is the lack of a structured regression testing process. In this regard, checklists can help practitioners keep track of essential regression testing activities and add structure to the regression testing process to a certain extent. This study aims to introduce regression testing checklists so test managers/teams can use them: (1) to assess whether test teams/members are ready to begin regression testing, and (2) to keep track of essential regression testing activities while planning and executing regression tests. We used interviews, workshops, and questionnaires to design, evolve, and evaluate regression testing checklists. In total, 25 practitioners from 12 companies participated in creating the checklist. Twenty-three of them participated in checklists evolution and evaluation. We identified activities practitioners consider significant while planning, performing, and analyzing regression testing. We designed regression testing checklists based on these activities to help practitioners make informed decisions during regression testing. With the help of practitioners, we evolved these checklists into two iterations. Finally, the practitioners provided feedback on the proposed checklists. All respondents think the proposed checklists are useful and customizable for their environments, and 80% think checklists cover aspects essential for regression testing. The proposed regression testing checklists can be useful for test managers to assess their team/team members’ readiness and decide when to start and stop regression testing. The checklists can be used to record the steps required while planning and executing regression testing. Further, these checklists can provide a basis for structuring the regression testing process in varying contexts.

Abstract Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large‐scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal–question–metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.

Abstract Replication studies help solidify and extend knowledge by evaluating previous studies’ findings. Software engineering literature showed that too few replications are conducted focusing on software artifacts without the involvement of humans. This study aims to replicate an artifact-based study on software testing to address the gap related to replications. In this investigation, we focus on (i) providing a step-by-step guide of the replication, reflecting on challenges when replicating artifact-based testing research and (ii) evaluating the replicated study concerning the validity and robustness of the findings. We replicate a test case prioritization technique proposed by Kwon et al. We replicated the original study using six software programs, four from the original study and two additional software programs. We automated the steps of the original study using a Jupyter notebook to support future replications. Various general factors facilitating replications are identified, such as (1) the importance of documentation; (2) the need for assistance from the original authors; (3) issues in the maintenance of open-source repositories (e.g., concerning needed software dependencies, versioning); and (4) availability of scripts. We also noted observations specific to the study and its context, such as insights from using different mutation tools and strategies for mutant generation. We conclude that the study by Kwon et al. is partially replicable for small software programs and could be automated to facilitate software practitioners, given the availability of required information. However, it is hard to implement the technique for large software programs with the current guidelines. Based on lessons learned, we suggest that the authors of original studies need to publish their data and experimental setup to support the external replications.

Code review is a common practice in software development and numerous studies have described different aspects of the process; its characteristics, the expectations on that process, issues around reviewer allocation, and more. However, one aspect that has not been studied to a large extent is the experience of the developers in the code review process. This is unfortunate given the significant amount of time that developers spend on this activity, where problems that degrade developers’ experience on a daily basis can create work environment issues.

Abstract Cross‐project defect prediction (CPDP), where data from different software projects are used to predict defects, has been proposed as a way to provide data for software projects that lack historical data. Evaluations of CPDP models using the Nearest Neighbour (NN) Filter approach have shown promising results in recent studies. A key challenge with defect‐prediction datasets is class imbalance, that is, highly skewed datasets where non‐buggy modules dominate the buggy modules. In the past, data resampling approaches have been applied to within‐projects defect prediction models to help alleviate the negative effects of class imbalance in the datasets. To address the class imbalance issue in CPDP, the authors assess the impact of data resampling approaches on CPDP models after the NN Filter is applied. The impact on prediction performance of five oversampling approaches (MAHAKIL, SMOTE, Borderline‐SMOTE, Random Oversampling and ADASYN) and three undersampling approaches (Random Undersampling, Tomek Links and One‐sided selection) is investigated and results are compared to approaches without data resampling. The authors examined six defect prediction models on 34 datasets extracted from the PROMISE repository. The authors' results show that there is a significant positive effect of data resampling on CPDP performance, suggesting that software quality teams and researchers should consider applying data resampling approaches for improved recall ( pd ) and g ‐measure prediction performance. However, if the goal is to improve precision and reduce false alarm ( pf ) then data resampling approaches should be avoided.

Behavior-driven development (BDD) is a variant of test-driven development where specifications are described in a structured domain-specific natural language. Although refactoring is a crucial activity of BDD, little research is available on the topic. To support practitioners in refactoring BDD specifications by (1) proposing semi-automated approaches to identify refactoring candidates; (2) defining refactoring techniques for BDD specifications; and (3) evaluating the proposed identification approaches in an industry context. Using Action Research, we have developed an approach for identifying refactoring candidates in BDD specifications based on two measures of similarity and applied the approach in two projects of a large software organization. The accuracy of the measures for identifying refactoring candidates was then evaluated against an approach based on machine learning and a manual approach based on practitioner perception. We proposed two measures of similarity to support the identification of refactoring candidates in a BDD specification base; (1) normalized compression similarity (NCS) and (2) similarity ratio (SR). A semi-automated approach based on NCS and SR was developed and applied to two industrial cases to identify refactoring candidates. Our results show that our approach can identify candidates for refactoring 6o times faster than a manual approach. Our results furthermore showed that our measures accurately identified refactoring candidates compared with a manual identification by software practitioners and outperformed an ML-based text classification approach. We also described four types of refactoring techniques applicable to BDD specifications; merging candidates, restructuring candidates, deleting duplicates, and renaming specification titles. Our results show that NCS and SR can help practitioners in accurately identifying BDD specifications that are suitable candidates for refactoring, which also decreases the time for identifying refactoring candidates.

Context: Modern software development increasingly relies on software testing for an ever more frequent delivery of high quality software. This puts high demands on the quality of the central artifacts in software testing, test suites and test cases. Objective: We aim to develop a comprehensive model for capturing the dimensions of test case/suite quality, which are relevant for a variety of perspectives. Methods: We have carried out a systematic literature review to identify and analyze existing secondary studies on quality aspects of software testing artifacts. Results: We identified 49 relevant secondary studies. Of these 49 studies, less than half did some form of quality appraisal of the included primary studies and only 3 took into account the quality of the primary study when synthesizing the results. We present an aggregation of the context dimensions and factors that can be used to characterize the environment in which the test case/suite quality is investigated. We also provide a comprehensive model of test case/suite quality with definitions for the quality attributes and measurements based on findings in the literature and ISO/IEC 25010:2011. Conclusion: The test artifact quality model presented in the paper can be used to support test artifact quality assessment and improvement initiatives in practice. Furthermore, the model can also be used as a framework for documenting context characteristics to make research results more accessible for research and practice.

The performance of software defect prediction(SDP) models is known to be dependent on the datasets used for training the models. Evolving data in a dynamic software development environment such as significant refactoring and organizational changes introduces new concept to the prediction model, thus making improved classification performance difficult. In this study, we investigate and assess the existence and impact of concept drift on SDP performances. We empirically asses the prediction performance of five models by conducting cross-version experiments using fifty-five releases of five open-source projects. Prediction performance fluctuated as the training datasets changed over time. Our results indicate that the quality and the reliability of defect prediction models fluctuate over time and that this instability should be considered by software quality teams when using historical datasets. The performance of a static predictor constructed with data from historical versions may degrade over time due to the challenges posed by concept drift.

Requirements prioritization plays a vital role in software release planning, and the process requires good interaction between stakeholders. Requirement prioritization process is challenging in global software development (GSD) context, where communication and coordination are core bottlenecks. Based on the issues found in the literature and industry survey, this paper proposes a requirements prioritization technique for GSD settings based on stakeholders' weight, vote, and priority (SWVP). The technique has two phases: (1) a calculation of stakeholders' profile weights and setting up communication parameters and (2) a voting process supporting the actual prioritization, based on the weights from phase (1). The proposed technique was applied to three industrial projects. The feedback of the practitioners after using SWVP was encouraging, and the feedback results indicate that SWVP could be a practical solution for requirements prioritization in GSD contexts.

Large-scale distributed software projects with long life cycles often involve a considerable amount of complex legacy code. The combination of scale and distribution challenges and the difficulty in acquiring knowledge about massive amounts of complex legacy code may make the onboarding of new developers/teams problematic. These problems may lead to extended periods of low performance. The primary objective of this paper is to investigate the performance evolution of offshore newcomers onboarded in a large-scale globally distributed project and how it relates to the employed onboarding strategy. To achieve our objective, we conducted a case study in Ericsson. We identified that the following aspects in the onboarding strategy employed in the investigated case seem to be related to the unexpectedly low performance evolution: i) the distance to mentors; ii) the used formal training approach, which did not fit the sociocultural background of the newcomers; iii) allocation of large and distributed tasks in the early stages of the onboarding process; and iv) team instability. We conclude that the onboarding of newcomers in globally distributed projects must be planned well ahead and should consider avoiding the aspects mentioned above.

Abstract Software developing organizations strive to achieve flexibility to maintain a competitive advantage. There is no common understanding of what characterize flexibility for a software organization beyond the scope of the software product. Without a common understanding, it is difficult to evaluate the degrees of flexibility of software development approaches. The aim of this literature review is to collect attributes that characterize flexibility. The collected attributes are consolidated into a flexibility framework with 3 main attributes: properties of change, flexibility perspectives, and flexibility enablers. The resulting flexibility framework is then used to evaluate Agile and Lean practices. The evaluation shows that Agile and Lean practices address many flexibility attributes. However, some attributes are not addressed, such as infrastructure flexibility and strategic flexibility. On the basis of our evaluation, the classifications of flexibility attributes that we present in this paper could be used to aid software organization flexibility evaluation.

Code quality is a key issue in software development. The ability to develop software of high quality is therefore a key learning goal of computing programs. However, there are no universally accepted measures to assess the quality of code and current standards are consideredweak. Furthermore, there are many facets to code quality. Defining and explaining the concept of code quality is therefore a challenge faced by many educators. In this working group, we investigate the perceptions of code quality of students, teachers, and professional programmers. In particular, we are interested in the differences in views of code quality by students, educators, and professional programmers and which quality aspects they consider as more or less important. Furthermore, we are interested in which sources of information on code quality and its assessment are used by these groups. Eventually, this will help us to develop resources that can be used to broaden students' views on software quality.

In Agile Software Development (ASD) effort estimation plays an important role during release and iteration planning. The state of the art and practice on effort estimation in ASD have been recently identified. However, this knowledge has not yet been organized. The aim of this study is twofold: (1) To organize the knowledge on effort estimation in ASD and (2) to use this organized knowledge to support practice and the future research on effort estimation in ASD. We applied a taxonomy design method to organize the identified knowledge as a taxonomy of effort estimation in ASD. The proposed taxonomy offers a faceted classification scheme to characterize estimation activities of agile projects. Our agile estimation taxonomy consists of four dimensions: estimation context, estimation technique, effort predictors and effort estimate. Each dimension in turn has several facets. We applied the taxonomy to characterize estimation activities of 10 agile projects identified from the literature to assess whether all important estimation-related aspects are reported. The results showed that studies do not report complete information related to estimation. The taxonomy was also used to characterize the estimation activities of four agile teams from three different software companies. The practitioners involved in the investigation found the taxonomy useful in characterizing and documenting the estimation sessions.

Context: Software Engineering (SE) is an evolving discipline with new subareas being continuously developed and added. To structure and better understand the SE body of knowledge, taxonomies have been proposed in all SE knowledge areas. Objective: The objective of this paper is to characterize the state-of-the-art research on SE taxonomies. Method: A systematic mapping study was conducted, based on 270 primary studies. Results: An increasing number of SE taxonomies have been published since 2000 in a broad range of venues, including the top SE journals and conferences. The majority of taxonomies can be grouped into the following SWEBOK knowledge areas: construction (19.55%), design (19.55%), requirements (15.50%) and maintenance (11.81%). Illustration (45.76%) is the most frequently used approach for taxonomy validation. Hierarchy (53.14%) and faceted analysis (39.48%) are the most frequently used classification structures. Most taxonomies rely on qualitative procedures to classify subject matter instances, but in most cases (86.53%) these procedures are not described in sufficient detail. The majority of the taxonomies (97%) target unique subject matters and many taxonomy-papers are cited frequently. Most SE taxonomies are designed in an ad-hoc way. To address this issue, we have revised an existing method for developing taxonomies in a more systematic way. Conclusion: There is a strong interest in taxonomies in SE, but few taxonomies are extended or revised. Taxonomy design decisions regarding the used classification structures, procedures and descriptive bases are usually not well described and motivated.

Team projects are a way to expose students to conflicting project objectives, and “[t]here should be a strong real-world element … to ensure that the experience is realistic” [ACM/IEEE-CS 2015b]. Team projects provide an opportunity for students to put their education into practice and prepare them for their professional careers. The aim of this special issue, and the previous one, is to collect and share evidence about the state of practice of team projects in computing education and to help educators in designing and running team projects. The articles presented in the present issue cover the following topics: real projects for real clients, open source projects, multidisciplinary team projects, student and team assessment, and cognitive and psychological aspects of team projects.

Abstract Context: Global software development (GSD) projects can have distributed teams that work independently in different locations or team members that are dispersed. The various development settings in GSD can influence quality during product evolution. When evaluating quality using defects as a proxy, the development settings have to be taken into consideration. Objective: The aim is to provide a systematic method for supporting investigations of the implication of GSD contexts on defect data as a proxy for quality. Method: A method engineering approach was used to incrementally develop the proposed method. This was done through applying the method in multiple industrial contexts and then using lessons learned to refine and improve the method after application. Results: A measurement instrument and visualization was proposed incorporating an understanding of the release history and understanding of GSD contexts. Conclusion: The method can help with making accurate inferences about development settings because it includes details on collecting and aggregating data at a level that matches the development setting in a GSD context and involves practitioners at various phases of the investigation. Finally, the information that is produced from following the method can help practitioners make informed decisions when planning to develop software in comparable circumstances. Copyright © 2016 John Wiley & Sons, Ltd.

Software readability and comprehension are important factors in software maintenance. There is a large body of research on software measurement, but the actual factors that make software easier to read or easier to comprehend are not well understood. In the present study, we investigate the role of method chains and code comments in software readability and comprehension. Our analysis comprises data from 104 students with varying programming experience. Readability and comprehension were measured by perceived readability, reading time and performance on a simple cloze test. Regarding perceived readability, our results show statistically significant differences between comment variants, but not between method chain variants. Regarding comprehension, there are no significant differences between method chain or comment variants. Student groups with low and high experience, respectively, show significant differences in perceived readability and performance on the cloze tests. Our results do not show any significant relationships between perceived readability and the other measures taken in the present study. Perceived readability might therefore be insufficient as the sole measure of software readability or comprehension. We also did not find any statistically significant relationships between size and perceived readability, reading time and comprehension.

Context: There are numerous studies on effort estimation in Agile Software Development (ASD) and the state of the art in this area has been recently documented in a Systematic Literature Review (SLR). However, to date there are no studies on the state of the practice in this area, focusing on similar issues to those investigated in the above-mentioned SLR. Objectives: The aim of this paper is to report on the state of the practice on effort estimation in ASD, focusing on a wide range of aspects such as the estimation techniques and effort predictors used, to name a few. Method: A survey was carried out using as instrument an on-line questionnaire answered by agile practitioners who have experience in effort estimation. Results: Data was collected from 60 agile practitioners from 16 different countries, and the main findings are: 1) Planning poker (63%), analogy (47%) and expert judgment (38%) are frequently practiced estimation techniques in ASD; 2) Story points is the most frequently (62%) employed size metric, used solo or in combination with other metrics (e.g., function points); 3) Team's expertise level and prior experience are most commonly used cost drivers; 4) 52% of the respondents believe that their effort estimates on average are under/over estimated by an error of 25% or more; 5) Most agile teams take into account implementation and testing activities during effort estimation; and 6) Estimation is mostly performed at sprint and release planning levels in ASD. Conclusions: Estimation techniques that rely on experts' subjective assessment are the ones used the most in ASD, with effort underestimation being the dominant trend. Further, the use of multiple techniques in combination and story points seem to present a positive association with estimation accuracy, and team-related cost drivers are the ones used by most agile teams. Finally, requirements and management related issues are perceived as the main reasons for inaccurate estimates.

Effort estimation is a project management activity that is mandatory for the execution of software projects. Despite its importance, there have been just a few studies published on such activities within the Agile Global Software Development (AGSD) context. Their aggregated results were recently published as part of a secondary study that reported the state of the art on effort estimation in AGSD. This study aims to complement the above-mentioned secondary study by means of an empirical investigation on the state of the practice on effort estimation in AGSD. To do so, a survey was carried out using as instrument an on-line questionnaire and a sample comprising software practitioners experienced in effort estimation within the AGSD context. Results show that the effort estimation techniques used within the AGSD and collocated contexts remained unchanged, with planning poker being the one employed the most. Sourcing strategies were found to have no or a small influence upon the choice of estimation techniques. With regard to effort predictors, global challenges such as cultural and time zone differences were reported, in addition to factors that are commonly considered in the collocated context, such as team experience. Finally, many challenges that impact the accuracy of the effort estimates were reported by the respondents, such as problems with the software requirements and the fact that the communication effort between sites is not properly accounted.

Team projects are a way to expose students to conflicting project objectives, and “[t]here should be a strong real-world element … to ensure that the experience is realistic” [ACM/IEEE-CS 2015b]. Team projects provide students an opportunity to put their education into practice and prepare them for their professional careers. The aim of this special issue is to collect and share evidence about the state-of-practice of team projects in computing education and to help educators in designing and running team projects. From a record number of 69 submitted abstracts, 19 were invited to submit a full paper. Finally, nine papers were accepted for publication in this and a subsequent issue. The articles presented in the present issue cover the following topics: real projects for real clients, open source projects, multidisciplinary team projects, student and team assessment, and cognitive and psychological aspects of team projects.

Example programs play an important role in the teaching and learning of programming. Students as well as teachers rank examples as the most important resources for learning to program. Example programs work as role models and must therefore always be consistent with the principles and rules we are teaching. However, it is difficult to find or develop examples that are fully faithful to all principles and guidelines of the object-oriented paradigm and also follow general pedagogical principles and practices. Unless students are able to engage with good examples, they will not be able to tell desirable from undesirable properties in their own and others’ programs. In this article we report on a study in which experienced educators evaluated the quality of object-oriented example programs for novices from popular Java textbooks. The evaluation was accomplished using an online checklist that elicited responses on the technical, object-oriented, and didactic quality of examples. In total 25 reviewers contributed 215 reviews to our dataset, based on 38 example programs from 13 common introductory programming textbooks. Results show that the evaluation instrument is reliable in terms of inter-rater agreement. Overall, example quality was not as good as one might expect from common textbooks, in particular regarding certain object-oriented properties. We conclude that educators should be careful when taking examples straight out of a textbook.

We have developed courseware for UML/SysML modelling based on the needs of the European embedded/automotive industry. The courseware supports interactive modelling exercises. First evaluations show promising results.

S.301

Example programs play an important role in learning to program. They work as templates, guidelines, and inspiration for learners when developing their own programs. It is therefore important to provide learners with high quality examples. In this paper, we discuss properties of example programs that might affect the teaching and learning of object-oriented programming. Furthermore, we present an evaluation instrument for example programs and report on initial experiences of its application to a selection of examples from popular introductory programming textbooks.

Expert judgment is a common estimation approach in industry. However, there is very little research on the accuracy of expert judgment outside the area of effort estimation. In this paper, we present an industrial case study investigating subjective and objective measures of requirementss volatility. Data was collected in retrospect for all use cases of a medium-size software project. In addition, we determined subjective volatility by interviewing developers and managers of the project. Our data analysis show that structural measures perform better than expert judgment in estimating the total number of changes to use case based requirements. These results confirm results from a previous case study by the authors and suggest that project managers should not rely on expert judgment alone for decision making.

Team project courses are important elements in most computer science and software engineering programs. For many students, the team project course represents the only non-trivial software development experience before graduation. The team project should be used to introduce them to important project and process issues that otherwise are very difficult to teach. Sloppy documentation, poor project planning and tracking or ineffective communication will eventually affect the teams and teach the students a important lessons for their future work. In this paper, we investigate how students select, carry out and complete their projects. The results show that students tend to select mainstream projects with good specifications and that certain project types are less suitable for the course. The results also show that process related deliverables are crucial to the final outcome of the projects. Among the hardest, but also most important, deliverables we find the project plan

Requirements volatility is an important risk factor for software projects. Software measures can help in quantifying and predicting this risk. In this paper, we present an industrial case study that investigated measures of volatility for a medium size software project. The goal of the study was twofold: 1) to empirically validate a set of measures associated with the volatility of use case models (UCM); 2) to investigate the correlation between subjective and objective volatility. Measurement data was collected in retrospect for all use case models of the software project. In addition, we determined subjective volatility by interviewing stakeholders of the project. Our data analysis showed a high correlation between our measures of size of UCM and total number of changes, indicating that the measures of size of UCMs are good indicators of requirements volatility. No correlations was found between subjective and objective volatility. These results suggest that project managers at this company should measure their projects because of the risk to take wrong decisions based on their own and the developer's perceptions.

The PLUSS approach (Product Line Use case modeling for Systems and Software engineering) is a domain modeling method tailored towards the development of long lived software intensive systems. PLUSS provides means to maintain a common and complete use case model for a whole family of systems. In this paper, we describe how the commercial requirements management tool Telelogic DOORS and the UML modeling tool IBM-Rational Rose can be extended and used for managing system family models in accordance with the PLUSS approach.

No abstract available.

Good communication skills are very important for any professional career. But few computer science curricula provide courses to acquire these skills. This paper describes a "conference course," which was held at our department to address research, technical writing and presentation skills. We outline the experiment, discuss our experiences, and present a proposal for a follow--up course. 1 Introduction Usually computer science curricula pay little attention to the development of communication skills. This is easy to understand, since the list of "need--to--know" topics still keeps growing. We squeeze in more and more knowledge units into our curricula to respond to the latest developments. But it is not only technical knowledge that makes a good computer scientist or software engineer. Without good communication skills the student's abilities cannot be fully developed. This is true not only for the computer science professional, but also for the researcher who needs to communicate h...

Modern computer-aided software engineering (CASE) tools raise the need for traceability, i.e. the ability to control the consistency between software documents produced at different stages of the software life-cycle. The authors describe the transformation between an integrated requirements engineering language based on structured analysis and the entity relationship model, and a modern design language. The transformation works incrementally and is sensitive to changes to already transformed parts. The authors outline the transformation algorithm and compare the approach to those found in the literature. A tool which implements the proposed algorithm and supports the proposed traceability is described. The potential and the limits of this approach are highlighted.>

The compression of bottom-up tree automata and their representation as implemented in the OPTRAN tree transformation system are described here as a four-step process.First, the vertically working tree automata traversing one generation in one step are replaced by horizontally working automata, splitting this step into rank (operator) steps, This replaces each matrix of dimension rank (operator) by a tree of depth rank (operator), A second step replaces this tree by a shortened directed acyclic graph (DAG), that is, a DAG where equivalent states of the tree are identified and paths in the tree with no gain in information are condensed, The acceptance property of the automaton is not changed.This and the following size reduction steps critically depend on the "no-error" assumption, that is, the assumption that input trees are correctly built and that no error detection is required of the tree automata.The third step embeds the DAG, which is naturally represented by two-dimensional tables, into a linear array using row displacement and row column schemes.In addition, this step compresses strings of consecl~.,veidentical entries, which occur often in this type of automaton.The last step exploits the memory structure and addressability of the target machine.Using information on automata and table sizes, the most efilcient storage representation is selected.The data structures and access functions are generated as C objects and functions, respectively.Results showing the effectiveness of the different steps are given.