This research to practice full paper observed that graduate subjects (or courses) are commonly offered in graduate programs and can provide specialized knowledge of different topics that are important for the formation of Ph.D. and Master’s students. At the same time, Systematic Literature Review (SLR) has been increasingly adopted in the computing area as a research method to synthesize the state of the art of a given research topic, identify research groups working on that topic, understand the existing limitations and research gaps, and also identify new research directions. However, it is still not well understood the real benefits and challenges of offering a subject that addresses SLR for graduate students. Moreover, it is not known the difficulties faced by professors (i.e., educators) to teach this subject. The main goal of this paper is to present an experience report of teaching SLR, in particular, the benefits and challenges of this subject for computer science graduate students. In addition, this paper also presents the essential topics of SLR that we recommend to be taught and a better way to teach them, aiming at supporting graduate courses to offer it. For this, we surveyed computer science graduate students who attended the SLR subject that was taught for almost ten years in our institutions. In particular, we collected the lessons learned, findings, and insights; following, we summarized the benefits and challenges for students, the difficulties for professors, and also those essential topics to be taught. As a main result, the SLR subject can be considered a valuable opportunity for graduate students that could use this subject to conduct the required deep literature review of their research topic and have a better comprehension of their research area. Besides and more importantly, this subject can improve important research skills of students, including the ability to recognize research problems, analyze and synthesize data, think critically, and write papers. Therefore, we believe that graduate courses should analyze the possibility of offering the SLR subject.

Context: For thousands of years humans have been using natural language to register their knowledge on important information to enable its access to future generations. With internet, a large amount of textual data is produced and shared on a daily basis. So, scientists started to research techniques for efficiently process knowledge stored in textual format. In this context, Natural Language Processing (NLP) became a popular area studying linguistic phenomena and using computational methods to process texts in natural language. In particular, Open Information Extraction (Open IE) was proposed to gather information from plain text. Despite the advances in this area, it is still necessary to map details about how these approaches were proposed to support the community while creating more efficient Open IE systems. Objective: In this paper, we identify, in the literature, the main characteristics of proposed Open IE approaches. Method: First, we extended the search performed in a systematic mapping previously published by using backward snowballing and a manual search. Next, we updated the electronic database search including ACL Anthology. Finally, 159 studies proposing Open IE approaches were considered for data extraction. Results: Data analysis showed a significant increase in the number of studies published about Open IE in the last years. In addition, we provide important details about how these techniques were proposed (e.g., data sets used and output evaluation techniques). Results indicate that researchers started to adopt neural networks to perform Open IE instead of using conventional supervised learning techniques. Conclusion: Recent advances in Artificial Intelligence and neural networks techniques allowed scientists to have a new perspective on how to perform efficient textual data management. Therefore, Open IE approaches gained much attention as they can help in many contexts, especially in knowledge management tasks.

Background: The software engineering community has increasingly conducted systematic literature reviews (SLR) as a means to summarize evidence from different studies and bring to light the state of the art of a given research topic. While SLR provide many benefits, they also present several problems with punctual solutions for some of them. However, two main problems still remain: the high time-/effort-consumption nature of SLR and the lack of an effective impact of SLR results in the industry, as initially expected for SLR. Aims: The main goal of this paper is to introduce a new view - which we name Sustainability of SLR - on how to deal with SLR aiming at reducing those problems. Method: We analyzed six reference studies published in the last decade to identify, group, and analyze the SLR problems and their interconnections. Based on such analysis, we proposed the view of Sustainability of SLR that intends to address these problems. Results: The proposed view encompasses three dimensions (social, economic, and technical) that could become SLR more sustainable in the sense that the four major problems and 31 barriers (i.e., possible causes for those problems) that we identified could be mitigated. Conclusions: The view of Sustainability of SLR intends to change the researchers' mindset to mitigate the inherent SLR problems and, as a consequence, achieve sustainable SLR, i.e., those that consume less time/effort to be conducted and updated with useful results for the industry.

Context: A tertiary study can be performed to identify related reviews on a topic of interest. However, the elaboration of an appropriate and effective search string to detect secondary studies is challenging for Software Engineering (SE) researchers. Objective: The main goal of this study is to propose a suitable search string to detect secondary studies in SE, addressing issues such as the quantity of applied terms, relevance, recall and precision. Method: We analyzed seven tertiary studies under two perspectives: (1) structure – strings’ terms to detect secondary studies; and (2) field: where searching – titles alone or abstracts alone or titles and abstracts together, among others. We validate our string by performing a twostep validation process. Firstly, we evaluated the capability to retrieve secondary studies over a set of 1537 secondary studies included in 24 tertiary studies in SE. Secondly, we evaluated the general capacity of retrieving secondary studies over an automated search using the Scopus digital library. Results: Our string was capable to retrieve an optimum value of over 90% of the included secondary studies (recall) with a high general precision of almost 60%. Conclusion: The suitable search string for finding secondary studies in SE contains the terms “systematic review”, “literature review”, “systematic mapping”, “mapping study” and “systematic map”.

Context: Secondary studies, as Systematic Literature Reviews (SLRs) and Systematic Mappings (SMs), have been providing methodological and structured processes to identify and select research evidence in Computer Science, especially in Software Engineering (SE). One of the main activities of a secondary study process is to read the abstracts to decide on including or excluding studies. This activity is considered costly and time-consuming. In order to speed up the selection activity, some alternatives such as, structured abstracts and graphical abstracts (e.g. Concept Maps – CMs), have been proposed. Objective: This study presents an approach to automatically build CMs using Natural Language Processing (NLP) to support the selection activity of secondary studies. Method: First, we proposed an approach composed by two pipelines: (1) perform the triple extraction of concept-relation-concept based on NLP; and (2) attach the extracted triples in a structure used as a template to scientific studies. Second, we evaluated both pipelines conducting experiments. Results: The preliminary evaluation revealed that CMs extracted are coherent when compared with their source text. Conclusions: NLP can assist the automatic construction of CMs. In addition, the experiment results show that the approach can be useful to support researchers in the selection of studies in the selection activity of secondary studies.

Context: Systematic reviews (SRs) have been adopted in the Software Engineering (SE) field for more than a decade to provide synthesis of evidence on various topics. However, the process in conducting an SR remains laborious-intensive and expensive, specially in terms of hours that SR researchers dedicate. It is worth exploring approaches to conduct SRs at lower costs (quicker, using less resources - time of researchers). One such approach is crowdsourcing, since conducting SRs activities among a large number of researchers is a promising alternative to reduce costs associated to SR conduction. Goal: The main goal of this study is to identify and summarize the body of knowledge on crowdsourcing to support the conduction of SRs in SE. Method: Two empirical research methods were used. Initially, we conducted a Systematic Mapping to identify the available and relevant studies on crowdsourcing in SRs in SE. Secondly, a survey was performed with 39 SE researchers aiming to identify their perception related to the value of performing SRs collaboratively. Results: Our results show that how to speed up the SR process; reduce bias through broad participation; and expand team expertise were most potential benefits linked to the use of crowdsourcing in SR. The main challenges were associated with quality control to ensure the quality of results. Conclusions: In spite of the challenges, we believe that crowdsourcing could be successfully employed in SR context. More empirical research is needed on how to use crowdsourcing to support SR conduction in SE and how to minimize the identified challenges.

Context: Systematic Literature Reviews (SLRs) are important instruments for both Software Engineering (SE) practitioners and scientific community. Their value directly depends on their quality and up-to-date results. However, most of the SLRs are outdated and the current scenario on how SLRs are documented does not favor their updating process. Goal: In this scenario, the main goal of this paper is to present an experience report on how to transfer the know-how of SLRs to facilitate their updates. Method: To address this issue, we used a Knowledge Management (KM) model, known as Nonaka-Takeuchi model, and described how we instantiated the Model for SLR update. We use two SLRs updates conducted by us to illustrate some of the knowledge sharing issues. Results: Our examples showed that the introduction of the concept of KM in the SLR update is in fact valuable, especially for sharing tacit knowledge (decisions) taken throughout the review process. Conclusions: We conclude that KM principles can be applied to manage the knowledge generated during the update of SLR.

Context: Despite being defined during the first phase of the Systematic Literature Review (SLR) process, the protocol is usually refined when other phases are performed. Several researchers have reported their experiences in applying SLRs in Software Engineering (SE) however, there is still a lack of studies discussing the iterative nature of the protocol definition, especially how it should be perceived by researchers conducting SLRs. Objective: The main goal of this study is to perform a survey aiming to identify: (i) the perception of SE researchers related to protocol definition; (ii) the activities of the review process that typically lead to protocol refinements; and (iii) which protocol items are refined in those activities. Method: A survey was performed with 53 SE researchers. Results: Our results show that: (i) protocol definition and pilot test are the two activities that most lead to further protocol refinements; (ii) data extraction form is the most modified item. Besides that, this study confirmed the iterative nature of the protocol definition. Conclusions: An iterative pilot testcan facilitate refinements in the protocol.

Background: Several researchers have reported their experiences in applying secondary studies in Software Engineering (SE), however, there is a lack of studies discussing the distinction between Systematic Mappings (SMs) and Systematic Literature Reviews (SLRs).Aims: The objective of this paper is to present the results of a tertiary study conducted to collect and evaluate evidence to better understand similarities and differences between SLRs and SMs related to four aspects: research question, search string, search strategy and quality assessment.Method: We identified 170 secondary studies that were reviewed to answer a set of Research Questions (RQ) related to the practical conduction of secondary studies in SE.Results: Results show that both SLRs and SMs have generic RQs, broad search strings, and adopt automatic search as search strategy.However, quality assessment has been more widely adopted in SLRs.Conclusions: In practice, only the quality assessment is conducted differently in SLRs and SMs.

Background: A Systematic Literature Review (SLR) is a methodology used to aggregate relevant evidence related to one or more research questions. Whenever new evidence is published after the completion of a SLR, this SLR should be updated in order to preserve its value. However, updating SLRs involves significant effort. Objective: The goal of this paper is to investigate the application of forward snowballing to support the update of SLRs. Method: We compare outcomes of an update achieved using the forward snowballing versus a published update using the search-based approach, i.e., searching for studies in electronic databases using a search string. Results: Forward snowballing showed a higher precision and a slightly lower recall. It reduced in more than five times the number of primary studies to filter however missed one relevant study. Conclusions: Due to its high precision, we believe that the use of forward snowballing considerably reduces the effort in updating SLRs in Software Engineering; however the risk of missing relevant papers should not be underrated.