The evidence-based paradigm has made software engineering (SE) more objective and less imprecise, facilitating extensive decision-making activities in software projects. However, by referring to the four-theory classificatory scheme, it can be observed that evidence-based SE mainly focuses on empirically developing descriptive theories from SE practices, and directly using descriptions to make predictions and suggest prescriptions, without emphasising the establishment and utilisation of explanatory theories for comprehensively understanding SE phenomena. In keeping with the pioneering viewpoint that regards explainability as a first-class citizen of SE in the future, we urge the development of explainable SE as a dedicated research field that pursues the state of the art beyond the state of the practice of SE. To support this forward-looking vision, we have developed an extensible framework, two factorial strategies, and a set of augmented empirical methods as the preliminary responses to two root research questions what to explain and how to explain and one derived research question how to identify proper explanations. As such, although at a vision stage, our current work has paved the way forward towards a more explainable future of SE that will bring various benefits to different stakeholders in the software ecosystem.

The term and label, “case study”, is not used consistently by authors of primary studies in software engineering research. It is not clear whether this problem also occurs for systematic literature studies (SLSs). To investigate the extent to which SLSs in/correctly use the term and label, “case study”, when classifying primary studies. We systematically collect two sub-samples (2010–2021 & 2022) comprising a total of eleven SLSs and 79 primary studies. We examine the designs of these SLSs, and then analyse whether the SLS authors and the primary-study authors correctly label the respective primary study as a “case study”. 76% of the 79 primary studies are misclassified by SLSs (with the two sub-samples having 60% and 81% misclassification, respectively). For 39% of the 79 studies, the SLSs propagate a mislabelling by the original authors, whilst for 37%, the SLSs introduce a new mislabel, thus making the problem worse. SLSs rarely present explicit definitions for “case study” and when they do, the definition is not consistent with established definitions. SLSs are both propagating and exacerbating the problem of the mislabelling of primary studies as “case studies”, rather than – as we should expect of SLSs – correcting the labelling of primary studies, and thus improving the body of credible evidence. Propagating and exacerbating mislabelling undermines the credibility of evidence in terms of its quantity, quality and relevance to both practice and research.

Covid-19 pandemic brought dramatic changes to higher education settings, particularly for curriculum delivery, moving quickly to online learning. This paper discusses teaching experience covering practices of technology-enhanced learning for the MSc Computing Foundations module (20 CATS) for a large class studying part-time Software Engineering course at the School of Electronics, Electrical Engineering and Computer Science (EEECS), Queen's University Belfast (QUB) during 2019-2022 academic years. We compare on-campus learning with the abrupt shift to online learning during the pandemic and with sustainable online learning a year later. The objective of this study is to answer how part-time Software Development students perceived their technology-enhanced learning experience from pre- to post-pandemic and to evaluate the impact of the shift to online learning for the part-time class This study is based on data collected during 2019/2020, 2020/2021, 2021/2022 academic years. Methodology types employed in this study include online observation with statistics collected from the virtual learning environment (VLE) Canvas, quantitative analysis, individual student surveys on teaching techniques and module content This study provided an effective online teaching method for Computer Foundation module, reviewing the impact of different curriculum items and online educational activities starting with content delivery – both synchronous and asynchronous – and moving on to VLE Canvas discussion forums, ungraded formative quizzes, in-term formative assessment in the form of mock exam and, finally, to online summative assessment delivered on VLE Canvas. We investigate positive and negative aspects of technology-enhanced learning from pre- to post-pandemic according to part-time adult students studying the MSc Software Development program and focus on how this effective learning environment contributes to education practice with a view what developments are worth retaining post-pandemic and what did not work well. Analysis of the data from individual student surveys on teaching techniques and module content for the Computing Foundations module allowed us to conclude that students perceived very positive their technology-enhanced learning experience after shifting to online learning. We also found out that сhanging the module delivery format (from face-to-face to online) did not affect the results of the students’ performance. Adaptation of the MSc Computing Foundation module to a new model of distance learning has proved to be successful, so we can conclude that this delivery format are appropriate for this target audience. This study explored the effectiveness of the pedagogical approach while also gaining valuable insights into the software development student experience of learning in the VLE. The findings from this study may contribute to developing effective teaching practices in software engineering education and adult learning, and improve the preparation of future software professionals in IT industry.

[Context]: In the university curriculum, software engineering (SE) is frequently perceived as a difficult course to study due to its concept-intensive nature. [Objectives]: We aim to investigate if and how we can help students conveniently understand and memorise the numerous and various SE concepts. [Method]: We employ critical reflection as the research method to qualitatively examine our educational activities, teaching materials, and students' learning effects. [Results]: By focusing on the paired structural model of knowledge representation, we propose to utilise tangentially-paired concepts together with the conventional bipolar-paired concepts to facilitate SE education. In particular, we have identified three types of tangentially-paired concepts with deriving (stemming), analogical, and cloning relationships respectively. [Conclusions]: Exploiting paired SE concepts can act as an efficient educational approach that supplements the existing strategies for teaching SE knowledge.

As one of the next frontiers of software engineering, agile development at scale has attracted more and more research interests and efforts. When following the existing autonomy-focused and goal-driven lessons and guidelines to scale agile development for a large astronomy project, however, we encountered surprising tech stack sprawl and spreading team coordination issues. By revisiting the unique features of our project (e.g., the data processing-intensive nature and the frequent team member changes), and by identifying a fractal pattern from various data processing logic and processes, we defined disciplined agile teams to clone the best practices of pioneer agile teams, and to work on similar system modules with similar user stories. Such a targeted strategy effectively relieved the tech stack sprawl and facilitated teamwork handover, at least for refactoring and growing the data processing modules in our project. Based on this emerging result and our reflections, we distinguish this targeted strategy as scaling out agile development from the existing agile scaling approaches that are generally in a scaling-up fashion. Considering the popularity of data processing-intensive projects, and also considering the pervasive fractal patterns in modern businesses and organisations, we claim that this targeted strategy still has broad application opportunities. Therefore, developing a well-defined methodology for scaling out agility, and combining both scaling up and scaling out agility, will deserve attentions and new research efforts in the future.

The definition and term “case study” are not being applied consistently by software engineering researchers. We previously developed a trivial “smell indicator” to help detect the misclassification of primary studies as case studies. To evaluate the performance of the indicator. We compare the performance of the indicator against human classifiers for three datasets, two datasets comprising classifications by both authors of systematic literature studies and primary studies, and one dataset comprising only primary-study author classifications. The indicator outperforms the human classifiers for all datasets. The indicator is successful because human classifiers “fail” to properly classify their own, and others’, primary studies. Consequently, reviewers of primary studies and authors of systematic literature studies could use the classifier as a “sanity” check for primary studies. Moreover, authors might use the indicator to double-check how they classified a study, as part of their analysis, and prior to submitting their manuscript for publication. We challenge the research community to both beat the indicator, and to improve its ability to identify true case studies.

Cloud computing has made federated database systems (FDBS) significantly more practical to implement than in the past. As part of a recent Web-based Geographic Information System (WebGIS) project, we are employing cloud-native technologies (from the container ecosystem) to develop a federated database (DB) infrastructure, to help manage and utilise the distributed and various geospatial data. Unfortunately, there seem to be inherent challenges and complexity of applying the container and Kubernetes technologies to building and running DB systems. Considering that most of the geospatial and theme data are pre-obtained and fixed in our WebGIS project, we decided to focus on the read-only user queries and still resort to Kubernetes to implement an FDBS instance to use. Unlike the de facto practices (e.g., using the StatefulSets mechanism, extending Kuberentes APIs, or employing KubeFed), our solution for Kubernetes-aided FDBS simplifies the tech stack by investigating the fractal object of federated data management, inclusively containerising DB instances, and using the lightweight Deployment mechanism to handle stateless DB containers. Overall, this research not only reveals an easy-to-implement approach to constructing read-only components in a fully-fledged FDBS, but also proposes and demonstrates a novel methodology for FDBS investigations.

Software practitioners are a primary provider of information for field studies in software engineering. Research typically recruits practitioners through some kind of sampling. But sampling may not in itself recruit the “right” participants. To assess existing guidance on participant recruitment, and to propose and illustrate a framework for recruiting professional practitioners as credible participants in field studies of software engineering. We review existing guidelines, checklists and other advisory sources on recruiting participants for field studies. We develop a framework, partly based on our prior research and on the research of others. We search for and select three exemplar studies (a case study, an interview study and a survey study) and use those to illustrate the framework. Whilst existing guidance recognises the importance of recruiting participants, there is limited guidance on how to recruit the “right” participants. The framework suggests the conceptualisation of participants as “research instruments” or, alternatively, as a sampling frame for items of interest. The exemplars suggest that at least some members of the research community are aware of the need to carefully recruit the “right” participants. The framework is intended to encourage researchers to think differently about the involvement of practitioners in field studies of software engineering. Also, the framework identifies a number of characteristics not explicitly addressed by existing guidelines.

The term “case study” is not used consistently when describing studies and, most importantly, is not used according to the established definitions. Given the misuse of the term “case study”, we critically analyse articles that cite case study guidelines and report case studies. We find that only about 50% of the studies labelled “case study” are correctly labelled, and about 40% of studies labelled “case study” are actually better understood as “small-scale evaluations”. Based on our experiences conducting the analysis, we formulate support for ensuring and assuring the correct labelling of case studies. We develop a checklist and a self-assessment scheme. The checklist is intended to complement existing definitions and to encourage researchers to use the term “case study” correctly. The self-assessment scheme is intended to help the researcher identify when their empirical study is a “small-scale evaluation” and, again, encourages researchers to label their studies correctly. Finally, we develop and evaluate a smell indicator to automatically suggest when a reported case study may not actually be a case study. These three instruments have been developed to help ensure and assure that only those studies that are actually case studies are labelled as “case study”.

BACKGROUND: Software engineering is a human activity. People naturally make sense of their activities and experience through storytelling. But storytelling does not appear to have been properly studied by software engineering research. AIM: We explore the question: what contribution can storytelling make to human–centric software engineering research? METHOD: We define concepts, identify types of story and their purposes, outcomes and effects, briefly review prior literature, identify several contributions and propose next steps. RESULTS: Storytelling can, amongst other contributions, contribute to data collection, data analyses, ways of knowing, research outputs, interventions in practice, and advocacy, and can integrate with evidence and arguments. Like all methods, storytelling brings risks. These risks can be managed. CONCLUSION: Storytelling provides a potential counter–balance to abstraction, and an approach to retain and honour human meaning in software engineering.

Background: Blogs are a source of grey literature which are widely adopted by software practitioners for disseminating opinion and experience. Analysing such articles can provide useful insights into the state–of–practice for software engineering research. However, there are challenges in identifying higher quality content from the large quantity of articles available. Credibility assessment can help in identifying quality content, though there is a lack of existing corpora. Credibility is typically measured through a series of conceptual criteria, with ’argumentation’ and ’evidence’ being two important criteria.

Context: An evidence-based scientific discipline should produce, consume and disseminate credible evidence. Unfortunately, mistakes are sometimes made, resulting in the production, consumption and dissemination of invalid or otherwise questionable evidence. In the worst cases, such questionable evidence achieves the status of accepted knowledge. There is, therefore, the need to ensure that producers and consumers seek to identify and rectify such situations. Objectives: To raise awareness of the negative impact of misinterpreting evidence and of propagating that misinterpreted evidence, and to provide guidance on how to improve on the type of issues identified. Methods: We use a case-based approach to present and analyse the production, consumption and dissemination of evidence. The cases are based on the literature and our professional experience. These cases illustrate a range of challenges confronting evidence-based researchers as well as the consequences to research when invalid evidence is not corrected in a timely way. Results: We use the cases and the challenges to formulate a framework and a set of recommendations to help the community in producing and consuming credible evidence. Conclusions: We encourage the community to collectively remain alert to the emergence and dissemination of invalid, or otherwise questionable, evidence, and to proactively seek to identify and rectify it.

Retrieving and mining practitioners’ self-reports of their professional experience of software practice could provide valuable evidence for research. The authors are, however, unaware of any existing reviews of research conducted in this area. The authors reviewed and classified previous research, and identified insights into the challenges research confronts when retrieving and mining practitioners’ self-reports of their experience of software practice. They conducted an exploratory review to identify and classify 42 studies. They analysed a selection of those studies for insights on challenges to mining professional experience. They identified only one directly relevant study. Even then this study concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. They discussed the challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. No directly relevant prior research appears to have been conducted in this area. They discussed the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. They formulated a set of guiding questions for further research in this area.

Background: Blog posts offer potential benefits for research, but also present challenges. The use of blog posts in SE research is contentious for some members of the community. Also, there are no guidelines for evaluating the credibility of blog posts.

Background: Software engineering (SE) research continues to study the degree to which practitioners perceive research as relevant to practice. Such studies typically comprise surveys of practitioner opinions. In a preliminary, and relatively small scale, study of online articles we previously found few explicit citations to software testing research. Our previous study provided an in situ complement to the typical survey study, however the findings of the previous study were limited by the size of our sample.

Abstract Background An emerging body of research is using grey literature to investigate software practice. One frequently occurring type of grey literature is the blog post. Whilst there are prospective benefits to using grey literature and blog posts to investigate software practice, there are also concerns about the quality of such material. Objectives To identify and describe the benefits and challenges to using blog‐like content to investigate software practice, and to scope directions for further research. Methods We conduct a review of previous research, mainly within software engineering, to identify benefits, challenges, and directions and use that review to complement our experiences of using blog posts in research. Results and Conclusion We identify and organise benefits and challenges of using blog‐like documents in software engineering research. We develop a definition of the type of blog‐like document that should be of (more) value to software engineering researchers. We identify and scope several directions in which to progress research into and with blog‐like documents. We discuss similarities and differences in secondary and primary studies that use blog‐like documents and similarities and differences between the use of blog‐like documents and the use of already established research methods, eg, interview and survey.

Context: The validity of survey-based research depends on, amongst other considerations, the number and validity of obtained data points. As with any empirical study that involves practitioners, collecting data via surveys is difficult. Objectives: We report our experiences derived while conducting an industry survey on the impact of agile practices on software process quality. Method: After unsuccessfully trying to collect data with an online questionnaire, we used an interactive approach with posters at practitioners-focussed software engineering community events to aid data collection. Results: We present a list of lessons learnt. In particular, the poster-based data collection approach provided utility, for both gathering a large amount of responses and facilitating follow-up interactions with study participants. Conclusion: Our experiences in this work may help those facing challenges associated with obtaining responses from practitioners through the use of potentially complex questionnaires.

Context: Open Source Software (OSS) developers use mailing lists as their main forum for discussing the evolution of a project. However, the use of mailing lists by developers for decision-making has not received much research attention. Objective: We have explored this issue by studying developers' email discussions around Python Enhancement Proposals (PEPs). Method: Our dataset comprised 42,672 emails from six different mailing lists pertaining to PEP development. We performed multiple forms of analysis on these emails, involving both quantitative measures (e.g., frequency) and deeper analysis of specific PEP discussions (i.e., outlier analysis). Results: Out of three PEP types (Informational, Process and Standard Track), Standard Track PEPs attract a large amount of discussion (both in volume and average number of messages per proposal). Our study also identified specific PEP states and topics that generated a disproportionate amount of discussion. Conclusion: Our outcomes point to several opportunities for improving the management of an OSS team based on the knowledge generated from discussions. We have also identified several interesting avenues for future work such as identifying individuals or groups that present persuasive arguments during decision-making.

Background: Blog articles have potential value as a source of practitioner-generated evidence to complement already accepted sources of evidence in software engineering research e.g. interviews and surveys. To be valuable to research, a method for extracting the high quality articles from the vast quantity available needs to be developed. Objective: To better define the benefits and challenges, scope the problem, develop a set of criteria for evaluating blog articles to be used in the method, and propose research questions. Method: We conducted a two-phase pilot study, using a preliminary set of criteria, to explore the challenges of classifying blog articles. We analyse credibility criteria that have been used in previous research, and cross reference those criteria with previous research in evidence-based software engineering. Results: Based on our analysis, we decide that blog articles need to be rigorous, relevant, well written and experience based for them to be considered credible to researchers. Conclusion: Our work provides an overview of the problem domain, as well as presenting criteria and suggested measurements for these criteria. These can be used by others to find blog articles of potential value to their research.

Background: researchers have a limited understanding of how practitioners conceive of and use evidence. Objective: to investigate how to automatically identify practitioner arguments and evidence in a corpus of practitioner documents, and identify insights for further work. Method: we develop, apply and evaluate a preliminary process to identify practitioner arguments and factual stories, based on the presence of specific words, using a sample of 1,022 blog posts from a software practitioner's blog. Results: we identify unanswered questions relating to the process: selecting and scraping data, cleansing data, parsing components of arguments and stories, selecting the 'right' cases, and validating and interpreting the results. Conclusion: our work provides a foundation for more substantive research on identifying practitioners' evidence and arguments that, in turn, can support research in other areas e.g. evidence informed software practice.

Severely impaired children with Physical and Neurological Impairments (PNI) often have erratic test responses because of impairments. Very often even the binary (YES/NO) intention of a PNI child cannot be determined because responses are made at the wrong time and conflicting signals are sent. We propose that it is possible to determine intention using significant streaks of successful responses found in the noisy responses. We can use two measures among others to determine intention; the maximum streak size attained in a test run with a device, and the sum of the significant streaks in the test run. The maximum streak size measures consecutive successes and the sum of streaks gives an indication measure of re-attempts. This work is part of a larger study to increase accessibility of PNI children to a cognitive test through the use of new non-hand held devices that interact with a computer. Using the proposed two measures, we are able to compare more closely the performance of the less capable and the more capable PNI children. The results show that the children who are more capable re-attempt in that test when they fail to achieve a target. Conversely the less able children divide into two groups: those that do re-attempt the target and those that do not re-attempt the target dependent however on the device being used. These results provide two measures that are potentially useful for determining the intention of the child undertaking the cognitive test.

The paper describes a mathematical model, the Exposure Model, for the prediction of customer reported defects in a large software system. In the model, exposure is defined as the likely fraction of the original defects in the software system that is reported by any customer, in any month and against any version of the system. The basic idea is to try to model exposure as the product of the isolated effects of a few customer- and system-characteristics. The model has been used for several purposes: to better understand defect detection mechanisms, and to better predict the resources required for defect correction. Also, the model has enabled us to make an early estimate of product quality, and thereby give valuable input for other important purposes.

Anecdotal evidence from our work with software developers suggests that maintenance is a significant problem for software development companies. A problem that is absorbing increasing amounts of precious development effort. In parallel, software companies are increasingly applying process improvement principles to development problems. In this paper we discuss how maintenance is addressed in process improvement programmes. We look at how well maintenance is addressed by formal process models like CMM. We also present empirical evidence from our study of process improvement in UK software companies. Our main findings are that although developers report that maintenance is indeed a problem, it is not always their most important problem. Furthermore, our findings also suggest that companies are often not well prepared for the maintenance phase of developments and that formal process improvement models do not pay enough attention to maintenance.