Background: The UK cyber skills gap/shortage amplifies the broader impact of cyber-attacks, which inflict harms such as privacy and economic loss on wider society. The demand is greatest (and growing fastest) in cyber-enabled disciplines, such as software engineering.

Background: Mining online accounts of practitioner experience can provide new evidence for software engineering researchers. However, we need methods for assessing quality at vast scale. Objectives: We investigate the challenge of finding high-quality grey literature, defining high-quality in terms of a document’s relevance to the consumer and its credibility. Method: Building on previous research, we use a version of the case survey methodology for automating the identification and assessment of high-quality grey literature. Results: We develop a model of credibility assessment within software engineering research and demonstrate our case survey methodology and credibility assessment model in practice. We use it to conduct a grey literature review of High Performing Teams (HPT). Conclusions: The paper provides a foundation for future research on automated quality and credibility assessment. Adoption of the tools and methodology presented can help researchers effectively search for and select higher-quality blog-like content.

Practitioner written grey literature, such as blog articles, has value in software engineering research. Such articles provide insight into practice that is often not visible to research. However, a high quantity and varying quality are two major challenges in utilising such material. Quality is defined as an aggregate of a document’s relevance to the consumer and its credibility. Credibility is often assessed through a series of conceptual criteria that are specific to a particular user group. For researchers, previous work has found ‘argumentation’ and ‘evidence’ to be two important criteria.

Background: Blogs are a source of grey literature which are widely adopted by software practitioners for disseminating opinion and experience. Analysing such articles can provide useful insights into the state–of–practice for software engineering research. However, there are challenges in identifying higher quality content from the large quantity of articles available. Credibility assessment can help in identifying quality content, though there is a lack of existing corpora. Credibility is typically measured through a series of conceptual criteria, with ’argumentation’ and ’evidence’ being two important criteria.

Retrieving and mining practitioners’ self-reports of their professional experience of software practice could provide valuable evidence for research. The authors are, however, unaware of any existing reviews of research conducted in this area. The authors reviewed and classified previous research, and identified insights into the challenges research confronts when retrieving and mining practitioners’ self-reports of their experience of software practice. They conducted an exploratory review to identify and classify 42 studies. They analysed a selection of those studies for insights on challenges to mining professional experience. They identified only one directly relevant study. Even then this study concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. They discussed the challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. No directly relevant prior research appears to have been conducted in this area. They discussed the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. They formulated a set of guiding questions for further research in this area.

Background: Blog posts offer potential benefits for research, but also present challenges. The use of blog posts in SE research is contentious for some members of the community. Also, there are no guidelines for evaluating the credibility of blog posts.

Background: Software engineering (SE) research continues to study the degree to which practitioners perceive research as relevant to practice. Such studies typically comprise surveys of practitioner opinions. In a preliminary, and relatively small scale, study of online articles we previously found few explicit citations to software testing research. Our previous study provided an in situ complement to the typical survey study, however the findings of the previous study were limited by the size of our sample.

Abstract Background An emerging body of research is using grey literature to investigate software practice. One frequently occurring type of grey literature is the blog post. Whilst there are prospective benefits to using grey literature and blog posts to investigate software practice, there are also concerns about the quality of such material. Objectives To identify and describe the benefits and challenges to using blog‐like content to investigate software practice, and to scope directions for further research. Methods We conduct a review of previous research, mainly within software engineering, to identify benefits, challenges, and directions and use that review to complement our experiences of using blog posts in research. Results and Conclusion We identify and organise benefits and challenges of using blog‐like documents in software engineering research. We develop a definition of the type of blog‐like document that should be of (more) value to software engineering researchers. We identify and scope several directions in which to progress research into and with blog‐like documents. We discuss similarities and differences in secondary and primary studies that use blog‐like documents and similarities and differences between the use of blog‐like documents and the use of already established research methods, eg, interview and survey.

Background: Blog articles have potential value as a source of practitioner generated evidence in grey literature reviews: they could complement already accepted sources (e.g. interviews and focus groups). To be valuable to research, blog articles need to be relevant, rigorous and evidence-based.

Background: Software engineering (SE) research continues to study the degree to which practitioners perceive that SE research has impact on practice. Such studies typically comprise surveys of practitioner opinions. These surveys could be complemented with other in situ practitioner sources e.g. grey literature.

Background: Blog articles have potential value as a source of practitioner-generated evidence to complement already accepted sources of evidence in software engineering research e.g. interviews and surveys. To be valuable to research, a method for extracting the high quality articles from the vast quantity available needs to be developed. Objective: To better define the benefits and challenges, scope the problem, develop a set of criteria for evaluating blog articles to be used in the method, and propose research questions. Method: We conducted a two-phase pilot study, using a preliminary set of criteria, to explore the challenges of classifying blog articles. We analyse credibility criteria that have been used in previous research, and cross reference those criteria with previous research in evidence-based software engineering. Results: Based on our analysis, we decide that blog articles need to be rigorous, relevant, well written and experience based for them to be considered credible to researchers. Conclusion: Our work provides an overview of the problem domain, as well as presenting criteria and suggested measurements for these criteria. These can be used by others to find blog articles of potential value to their research.

Background: researchers have a limited understanding of how practitioners conceive of and use evidence. Objective: to investigate how to automatically identify practitioner arguments and evidence in a corpus of practitioner documents, and identify insights for further work. Method: we develop, apply and evaluate a preliminary process to identify practitioner arguments and factual stories, based on the presence of specific words, using a sample of 1,022 blog posts from a software practitioner's blog. Results: we identify unanswered questions relating to the process: selecting and scraping data, cleansing data, parsing components of arguments and stories, selecting the 'right' cases, and validating and interpreting the results. Conclusion: our work provides a foundation for more substantive research on identifying practitioners' evidence and arguments that, in turn, can support research in other areas e.g. evidence informed software practice.