The rapidly growing number of scientific publications poses numerous challenges for researchers engaged in literature analyses. Structured methodologies like systematic literature reviews are becoming increasingly expensive, considering their attempt to cover all relevant publications. Despite the increasing efforts needed, the importance of literature reviews also leads to an increasing growth in their number. While there are support techniques (e.g., guidelines, tools, checklists) for conducting literature analyses, a concise and clear overview of such techniques for assessing the quality of the analysis itself is missing. Such an overview can help researchers identify techniques for their work, understand ambiguities between them, support peer reviews, and guide future research by highlighting open gaps. In this paper, we address this lack of an overview by identifying existing techniques for assessing the quality of systematic literature reviews, comparing their properties, and discussing their pros and cons. For this purpose, we elicited 14 techniques through a systematic literature search covering 15 years (2007–2021). Overall, our contributions can help researchers identify feasible techniques for assessing the quality of literature analyses and can guide the development of new techniques, thereby facilitating the conduct and improving the quality of literature analyses.

Academic research, by its nature, is notorious for being a challenging and demanding field. However, these challenges may become more complicated for certain groups of researchers rather than others. For instance, junior researchers who make up a large group of the current scientific community, particularly in the computer science domain, may face various types of impediments. A notable hindrance to realizing the impediments is the difficulty of precisely delineating them. In this paper, we report an empirical investigation to measure the level of awareness of any kind of obstacles that might hinder junior researchers’ publishing ability and disturb their involvement. For this purpose, we conducted a survey targeting active researchers from the Software Engineering field with a total of 52 respondents. We mainly focus on two types of aspects: peer reviewing models and collaboration. Our findings indicate that junior researchers seem to be more comfortable with double-blind reviewing models with more than half (approximately 67.2%) of them voting in favor of this model. The results also show a significant agreement that a lack of experience especially in academic writing and supervision problems constitute the most influential barriers to publishing. Our findings can help understand the needs of junior researchers and provide insights into our research community and its specific groups.

Background. The constantly increasing number of scientific publications poses challenges for researchers to monitor, select, and assess the publications relevant for their own research. Several guidelines for assessing publications manually during a literature analysis exist, with researchers proposing (semi-)automated techniques to facilitate such assessments. Aims. Still, research indicates that current techniques require further improvements to facilitate the analysis of large sets of publications. In this paper, we propose a semi-automatic technique with which we aim to improve in this direction by facilitating the selection and assessment of publications. Method. Our technique uses publicly available data of a publication, namely citation counts, article-level metrics, venue metrics, and altmetrics, to guide an analyst in assessing its relevance and impact. To evaluate the feasibility of our technique and the included metrics, we performed an experimental analysis to automatically assign ratings to the retrieved publications. Results. The results indicate that our technique can help an analyst in assessing publications, and reduce manual effort. Through our technique, we achieve an average accuracy of 53 % with a recall of 71 %. While precision (14 %) and F1-score (21 %) are—not surprisingly, due to the high number of irrelevant results returned by automatic searches in digital libraries—low, we see an improvement of these values for more recent reviews for which we could collect more complete data. However, some manual effort is still required for the final selection of papers. Conclusions. While it is not possible to achieve full automation for selecting and quality assessing publications, we can see that our metrics-based technique can be a helpful means to provide an initial rating for the analyst. Also, incorporating altmetrics seems to be a promising addition to rate comparably recent publications, helping researchers to further facilitate the execution of literature analyses.

The rapidly expanding corpus of scientific publications poses various types of challenges for researchers, mostly concerning the selection and assessment of publications relevant to their research topic. Therefore, the scientific community is actively involved in proposing solutions for effectively retrieving promising publications. Traditional bibliometrics, such as citations, are most commonly used for evaluating the research impact of a publication, in spite of rightful criticism. More recently, the newly introduced altmetrics (e.g., Tweets) have gained popularity and are constantly being investigated to understand their usefulness and potential benefits for assessing the significance of publications. Researchers argue that altmetrics can be used to reflect the importance of a publication beyond the boundaries of traditional bibliometrics. However, it is important to be aware of the limitations and threats arising from altmetrics, too. In this paper, we present a survey analysis to understand the usefulness of altmetrics and determine their ability of being used as quality indicators for scientific research. Based on the findings, we discuss whether altmetrics can support the quality assessment during literature analyses to assist the analyst by reducing the required time and manual effort.