Chong Zhang, Yi Tu, Yixi Zhao, Chenshu Yuan, Huan Chen, Yue Zhang, Mingxu Chai, Ya Guo, Huijia Zhu, Qi Zhang, Tao Gui. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Systematic reviews aim to provide high-quality evidence-based syntheses for efficacy under real-world conditions and allow understanding the correlations between exposures and outcomes. They are increasingly popular and have several stakeholders (e.g., healthcare providers, researchers, educators, students, journal editors, policy makers, managers) to whom they help make informed recommendations for practice or policy. Systematic reviews usually exhibit low methodological and reporting quality. To tackle this, reporting guidelines have been developed to support systematic reviews reporting and assessment. Following such guidelines is crucial to ensure that a review is transparent, complete, trustworthy, reproducible, and unbiased. However, systematic reviewers usually fail to adhere to existing reporting guidelines, which may significantly decrease the quality of the reviews they report and may result in systematic reviews that lack methodological rigor, yield low-credible findings and may mislead decision-makers. To assure that a review complies with reporting guidelines, we rely on assurance cases that are an emerging way of arguing and relaying various safety–critical systems’ requirements in an extensive manner, as well as checking the compliance of such systems with standards to support their certification. Since the nature of assurance cases makes them applicable to various domains and requirements/properties, we therefore propose a new type of assurance cases called systematicity cases. Systematicity cases focus on the systematicity property and allow arguing that a review is systematic i.e., that it sufficiently complies with the targeted reporting guideline. The most widespread reporting guidelines include PRISMA (Preferred Reporting Items for Systematic reviews and meta-Analyses). We measure the confidence in a systematicity case representing a review as a means to quantify the systematicity of that review i.e., the extent to which that review is systematic. We rely on rule-based Artificial Intelligence to create a knowledge-based system that automatically supports the inference mechanism that a given systematicity case embodies and that allows making a decision regarding the systematicity of a given review. An empirical evaluation performed on 25 reviews (self-identifying as systematic) showed that these reviews exhibit a suboptimal systematicity. More specifically, the systematicity of the analyzed reviews varies between 32.96% and 66.49% and its average is 54.42%. More efforts are therefore needed to report systematic reviews of higher quality. More experiments are also needed to further explore the factors hindering and/or assuring the systematicity of reviews. The main beneficiaries of our work are journal reviewers, journal editors, managers, policymakers, researchers, organizations developing reporting guidelines, peer reviewers, students, insurers, evidence users, as well as reporting guidelines developers.

Systematic reviews are crucial for various stakeholders since they allow them to make evidence-based decisions without being overwhelmed by a large volume of research. Systematic reviews are increasingly popular in the software engineering field. The abstract is one of the most important systematic review’s components since it usually reflects the content of the review. It may be the only part of the review that most of the readers will read when needing to form an opinion on a given topic. Besides, the content of an abstract is usually the main information readers use to decide if they want to access the full content of the review or not. Since an abstract usually summarizes a review, readers may therefore mostly rely on that abstract to judge the quality of the review as well as its methodological rigor. However, abstracts are sometimes poorly written and may therefore give a misleading and even harmful picture of the reviews’ contents. To assess abstracts, we propose a measure that allows quantifying the systematicity of reviews’ abstracts i.e., the extent to which these abstracts exhibit good reporting quality. Experiments on 151 reviews published in the software engineering (SE) field showed that these reviews’ abstracts exhibit a suboptimal systematicity.

Systematic reviews are crucial for various stakeholders since they allow them to make evidence-based decisions without being overwhelmed by a large volume of research. The abstract is one of the most important systematic review’s components since it usually reflects the content of the review. It may be the only part of the review that most of thereaders will read when needing to form an opinion on a given topic. Besides, the content of an abstract is usually the main information readers use to decide if they want to access the full content of the review or not. Since an abstract usually summarizes a review, readers may therefore mostly rely on that abstract to judge the quality of the review as well as its methodological rigor. However, abstracts are usually poorly written and may therefore give a misleading and even harmful picture of the reviews’ contents. To assess abstracts, we propose a measure that allows quantifying the systematicity of reviews’ abstracts i.e., the extent to which these abstracts exhibit good reporting quality. Experiments on 151 reviews published in the software engineering field showed that these reviews’ abstracts exhibit a suboptimal systematicity.