Software engineering courses enable practical learning through assignments requiring contributions to open source software (OSS), allowing students to experience real-world projects, collaborate with global communities, and develop skills and competencies required to succeed in the tech industry. Learning software engineering through open source contribution integrates theory with hands-on practice, as students tackle real challenges in collaborative environments. However, students often struggle to contribute to OSS projects and do not understand the contribution process. Research has demonstrated that strategically incorporating game elements can promote student learning and engagement. This paper proposes and evaluates OSSDoorway, a tool designed to guide students contributing to OSS projects. We recruited 29 students and administered a selfefficacy questionnaire before and after their use of OSSDoorway, along with qualitative feedback to assess challenges, interface features, and suggestions for improvement. The results show that OSSDoorway boosts students' self-efficacy and provides a structured, gamified learning experience. Clear instructions, real-time feedback, and the quest-based system helped students navigate tasks like using GitHub features to submit pull requests and collaborating with the community. Our findings suggest that providing students with a supportive gamified environment that uses feedback and structured quests can help them navigate the OSS contribution process.

Background: Systematic literature reviews (SLRs) require robust search strategies to ensure comprehensive coverage. Although database searches have traditionally been the primary method, snowballing has emerged as an effective alternative strategy in software engineering research. However, the success of snowballing heavily depends on the initial seed set's composition, particularly regarding diversity across authors, publication years, and venues. Objective: This study investigates how different diversity characteristics in seed set creation influence snowballing performance and effectiveness in identifying relevant literature. Method: We conducted replication studies of two existing SLRs, comparing their conventional seed set creation approaches with our diversity-driven methodology, where we systematically incorporated diversity characteristics into constructing the seed sets. Results: Our diversity-based approach demonstrated substantial improvements, with a precision of 0.019 (compared to 0.006 in the original), a relative recall of 0.97 (versus 0.921), and an F-measure of 0.0372 (improving from 0.0119). Conclusions: The empirical evidence suggests that incorporating diversity criteria in seed set creation enhances snowballing efficacy while maintaining comprehensive coverage of relevant literature. This approach offers a systematic and effective method for conducting snowballbased literature reviews in software engineering research.

Context: The Systematic Literature Review (SLR) process involves searching, selecting, and synthesizing relevant literature on a specific research topic for evidence-based decision-making in Software Engineering (SE). Due to the time-consuming of the SLR process, tool support is essential. Gap: ChatGPT is a significant advancement in Natural Language Processing (NLP), and it can potentially accelerate time-consuming and propone-error activities, such as the selection activity of the SLR process. Therefore, having a tool to assist in the selection process appears beneficial, and we argue that ChatGPT can facilitate the analysis of extensive studies, saving time and effort. Objective: We aim to evaluate the accuracy (i.e., studies correctly classified) of using ChatGPT–4.0 in SLR in SE, particularly to support the first stage, based on the title, abstract, and keywords. Method: We assessed the accuracy of utilizing ChatGPT for selecting studies, the first stage, to be included in two SLRs (SLR1 and SLR2), in contrast to the conventional method of reading the title and abstract. Results: The accuracy of ChatGPT supporting the initial selection activity was 75.3% (SLR1 – 101 correct selections: 48 inclusions and 53 exclusions; 33 incorrect selections: 17 inclusions and 16 exclusions) and 86.1% (SLR2 – 386 correct selections: 113 inclusions and 273 exclusions; 62 incorrect selections: 27 inclusions and 35 exclusions). Conclusions: Our accuracy results indicate that it is not advisable to completely outsource the selection process to ChatGPT. However, it could be valuable as a support tool, aiding novice researchers or even experienced ones when they are in doubt.

Context: Systematic mapping studies (SMS) are adopted in Software Engineering (SE) to select and synthesize relevant literature on a research topic and, thus, support evidence-based decision-making. Performing SMS is effort-demanding and time-consuming. Hence, using tools is beneficial. Large Language Models (LLMs) such as ChatGPT–4.o can potentially accelerate repetitive activities, such as data extraction in SMS, saving time and effort. Goal: We conducted this work to evaluate and provide preliminary evidence on how ChatGPT–4.o can support data extraction in SMS. Method: We performed a proof-of-concept study and assessed the results’ accuracy of using ChatGPT 4.0 to extract data in one SMS compared to the results produced manually. Results: The accuracy of ChatGPT–4.o was 87.83%. Conclusions: Our preliminary findings suggest that entirely replacing the manual data extraction with ChatGPT–4.o is not recommended. However, employing ChatGPT for semi-automated data extraction to aid in evidence synthesis in SMS is promising.

Context: There is a growing belief in the literature that large language models (LLMs), such as ChatGPT, can mimic human behavior in surveys. Gap: While the literature has shown promising results in social sciences and market research, there is scant evidence of its effectiveness in technical fields like software engineering. Objective: Inspired by previous work, this paper explores ChatGPT’s ability to replicate findings from prior software engineering research. Given the frequent use of surveys in this field, if LLMs can accurately emulate human responses, this technique could address common methodological challenges like recruitment difficulties, representational shortcomings, and respondent fatigue. Method: We prompted ChatGPT to reflect the behavior of a ‘mega-persona’ representing the demographic distribution of interest. We replicated surveys from 2019 to 2023 from leading SE conferences, examining ChatGPT’s proficiency in mimicking responses from diverse demographics. Results: Our findings reveal that ChatGPT can successfully replicate the outcomes of some studies, but in others, the results were not significantly better than a random baseline. Conclusions: This paper reports our results so far and discusses the challenges and potential research opportunities in leveraging LLMs for representing humans in software engineering surveys.

Contributing to OSS projects can help students to enhance their skills and expand their professional networks. However, novice contributors often feel discouraged due to various barriers. Gamification techniques hold the potential to foster engagement and facilitate the learning process. Nevertheless, it is unknown which game elements are effective in this context. This study explores students’ perceptions of gamification elements to inform the design of a gamified learning environment. We surveyed 115 students and segmented the analysis from three perspectives: (1) cognitive styles, (2) gender, and (3) ethnicity (Hispanic/LatinX and Non-Hispanic/LatinX). The results showed that Quest, Point, Stats, and Badge are favored elements, while competition and pressure-related are less preferred. Across cognitive styles (persona), gender, and ethnicity, we could not observe any statistical differences, except for Tim’s GenderMag persona, which demonstrated a higher preference for storytelling. Conversely, Hispanic/LatinX participants showed a preference for the Choice element. These results can guide tool builders in designing effective gamified learning environments focused on the OSS contributions process.

Context: There are several initiatives to semi-automate the initial selection of studies task for Systematic Literature Reviews (SLR) to reduce effort and potential bias. Objective: We propose a strategy called SCAS-AI to semi-automate the initial selection task. This strategy improves the original SCAS strategy with Artificial Intelligence (AI) resources (fuzzy logic and genetic algorithm) for studies selection. Method: We evaluated the SCAS-AI strategy through a quasi-experiment with SLRs in Software Engineering (SE). Results: In general, the SCAS-AI strategy improved the results achieved using the original SCAS strategy in reducing the effort of the initial selection task. The effort reduction applying SCAS-AI was 39.1%. In addition, the errors percentage was 0.3% for studies automatically excluded (false negative – loss of evidence) and 3.3% for studies automatically included (false positive – evidence later excluded during the full-text reading). Conclusion: The results show the potential of the investigated AI techniques to support the initial selection task for SLRs in SE.

This research to practice full paper observed that graduate subjects (or courses) are commonly offered in graduate programs and can provide specialized knowledge of different topics that are important for the formation of Ph.D. and Master’s students. At the same time, Systematic Literature Review (SLR) has been increasingly adopted in the computing area as a research method to synthesize the state of the art of a given research topic, identify research groups working on that topic, understand the existing limitations and research gaps, and also identify new research directions. However, it is still not well understood the real benefits and challenges of offering a subject that addresses SLR for graduate students. Moreover, it is not known the difficulties faced by professors (i.e., educators) to teach this subject. The main goal of this paper is to present an experience report of teaching SLR, in particular, the benefits and challenges of this subject for computer science graduate students. In addition, this paper also presents the essential topics of SLR that we recommend to be taught and a better way to teach them, aiming at supporting graduate courses to offer it. For this, we surveyed computer science graduate students who attended the SLR subject that was taught for almost ten years in our institutions. In particular, we collected the lessons learned, findings, and insights; following, we summarized the benefits and challenges for students, the difficulties for professors, and also those essential topics to be taught. As a main result, the SLR subject can be considered a valuable opportunity for graduate students that could use this subject to conduct the required deep literature review of their research topic and have a better comprehension of their research area. Besides and more importantly, this subject can improve important research skills of students, including the ability to recognize research problems, analyze and synthesize data, think critically, and write papers. Therefore, we believe that graduate courses should analyze the possibility of offering the SLR subject.

A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which was later complemented with snowball searches. Our conjecture is that we can perform even better searches if combining these two search approaches, referred to as a hybrid search strategy. Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared four alternative hybrid search strategies to assess whether we could identify more cost-efficient ways of searching for relevant primary studies. To compare and evaluate the hybrid search strategy, we replicated the search procedure in a systematic literature review (SLR) on industry–academia collaboration in software engineering. The SLR used a more “traditional” approach to searching for relevant articles for an SLR, while our replication was executed using a hybrid search strategy. In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30% more primary studies and even more studies when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when performing systematic literature studies. The hybrid search strategy is a strong contender for being used when performing systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.

Context: For thousands of years humans have been using natural language to register their knowledge on important information to enable its access to future generations. With internet, a large amount of textual data is produced and shared on a daily basis. So, scientists started to research techniques for efficiently process knowledge stored in textual format. In this context, Natural Language Processing (NLP) became a popular area studying linguistic phenomena and using computational methods to process texts in natural language. In particular, Open Information Extraction (Open IE) was proposed to gather information from plain text. Despite the advances in this area, it is still necessary to map details about how these approaches were proposed to support the community while creating more efficient Open IE systems. Objective: In this paper, we identify, in the literature, the main characteristics of proposed Open IE approaches. Method: First, we extended the search performed in a systematic mapping previously published by using backward snowballing and a manual search. Next, we updated the electronic database search including ACL Anthology. Finally, 159 studies proposing Open IE approaches were considered for data extraction. Results: Data analysis showed a significant increase in the number of studies published about Open IE in the last years. In addition, we provide important details about how these techniques were proposed (e.g., data sets used and output evaluation techniques). Results indicate that researchers started to adopt neural networks to perform Open IE instead of using conventional supervised learning techniques. Conclusion: Recent advances in Artificial Intelligence and neural networks techniques allowed scientists to have a new perspective on how to perform efficient textual data management. Therefore, Open IE approaches gained much attention as they can help in many contexts, especially in knowledge management tasks.

Background: The software engineering community has increasingly conducted systematic literature reviews (SLR) as a means to summarize evidence from different studies and bring to light the state of the art of a given research topic. While SLR provide many benefits, they also present several problems with punctual solutions for some of them. However, two main problems still remain: the high time-/effort-consumption nature of SLR and the lack of an effective impact of SLR results in the industry, as initially expected for SLR. Aims: The main goal of this paper is to introduce a new view - which we name Sustainability of SLR - on how to deal with SLR aiming at reducing those problems. Method: We analyzed six reference studies published in the last decade to identify, group, and analyze the SLR problems and their interconnections. Based on such analysis, we proposed the view of Sustainability of SLR that intends to address these problems. Results: The proposed view encompasses three dimensions (social, economic, and technical) that could become SLR more sustainable in the sense that the four major problems and 31 barriers (i.e., possible causes for those problems) that we identified could be mitigated. Conclusions: The view of Sustainability of SLR intends to change the researchers' mindset to mitigate the inherent SLR problems and, as a consequence, achieve sustainable SLR, i.e., those that consume less time/effort to be conducted and updated with useful results for the industry.

Context: A tertiary study can be performed to identify related reviews on a topic of interest. However, the elaboration of an appropriate and effective search string to detect secondary studies is challenging for Software Engineering (SE) researchers. Objective: The main goal of this study is to propose a suitable search string to detect secondary studies in SE, addressing issues such as the quantity of applied terms, relevance, recall and precision. Method: We analyzed seven tertiary studies under two perspectives: (1) structure – strings’ terms to detect secondary studies; and (2) field: where searching – titles alone or abstracts alone or titles and abstracts together, among others. We validate our string by performing a twostep validation process. Firstly, we evaluated the capability to retrieve secondary studies over a set of 1537 secondary studies included in 24 tertiary studies in SE. Secondly, we evaluated the general capacity of retrieving secondary studies over an automated search using the Scopus digital library. Results: Our string was capable to retrieve an optimum value of over 90% of the included secondary studies (recall) with a high general precision of almost 60%. Conclusion: The suitable search string for finding secondary studies in SE contains the terms “systematic review”, “literature review”, “systematic mapping”, “mapping study” and “systematic map”.

Context: Secondary studies, as Systematic Literature Reviews (SLRs) and Systematic Mappings (SMs), have been providing methodological and structured processes to identify and select research evidence in Computer Science, especially in Software Engineering (SE). One of the main activities of a secondary study process is to read the abstracts to decide on including or excluding studies. This activity is considered costly and time-consuming. In order to speed up the selection activity, some alternatives such as, structured abstracts and graphical abstracts (e.g. Concept Maps – CMs), have been proposed. Objective: This study presents an approach to automatically build CMs using Natural Language Processing (NLP) to support the selection activity of secondary studies. Method: First, we proposed an approach composed by two pipelines: (1) perform the triple extraction of concept-relation-concept based on NLP; and (2) attach the extracted triples in a structure used as a template to scientific studies. Second, we evaluated both pipelines conducting experiments. Results: The preliminary evaluation revealed that CMs extracted are coherent when compared with their source text. Conclusions: NLP can assist the automatic construction of CMs. In addition, the experiment results show that the approach can be useful to support researchers in the selection of studies in the selection activity of secondary studies.

Context: Systematic reviews (SRs) have been adopted in the Software Engineering (SE) field for more than a decade to provide synthesis of evidence on various topics. However, the process in conducting an SR remains laborious-intensive and expensive, specially in terms of hours that SR researchers dedicate. It is worth exploring approaches to conduct SRs at lower costs (quicker, using less resources - time of researchers). One such approach is crowdsourcing, since conducting SRs activities among a large number of researchers is a promising alternative to reduce costs associated to SR conduction. Goal: The main goal of this study is to identify and summarize the body of knowledge on crowdsourcing to support the conduction of SRs in SE. Method: Two empirical research methods were used. Initially, we conducted a Systematic Mapping to identify the available and relevant studies on crowdsourcing in SRs in SE. Secondly, a survey was performed with 39 SE researchers aiming to identify their perception related to the value of performing SRs collaboratively. Results: Our results show that how to speed up the SR process; reduce bias through broad participation; and expand team expertise were most potential benefits linked to the use of crowdsourcing in SR. The main challenges were associated with quality control to ensure the quality of results. Conclusions: In spite of the challenges, we believe that crowdsourcing could be successfully employed in SR context. More empirical research is needed on how to use crowdsourcing to support SR conduction in SE and how to minimize the identified challenges.

Context: Systematic Literature Reviews (SLRs) are important instruments for both Software Engineering (SE) practitioners and scientific community. Their value directly depends on their quality and up-to-date results. However, most of the SLRs are outdated and the current scenario on how SLRs are documented does not favor their updating process. Goal: In this scenario, the main goal of this paper is to present an experience report on how to transfer the know-how of SLRs to facilitate their updates. Method: To address this issue, we used a Knowledge Management (KM) model, known as Nonaka-Takeuchi model, and described how we instantiated the Model for SLR update. We use two SLRs updates conducted by us to illustrate some of the knowledge sharing issues. Results: Our examples showed that the introduction of the concept of KM in the SLR update is in fact valuable, especially for sharing tacit knowledge (decisions) taken throughout the review process. Conclusions: We conclude that KM principles can be applied to manage the knowledge generated during the update of SLR.

Systematic Literature Reviews (SLRs) have been adopted within Software Engineering (SE) for more than a decade to provide meaningful summaries of evidence on several topics. Many of these SLRs are now potentially not fully up-to-date, and there are no standard proposals on how to update SLRs in SE. The objective of this paper is to propose guidelines on how to best search for evidence when updating SLRs in SE, and to evaluate these guidelines using an SLR that was not employed during the formulation of the guidelines. To propose our guidelines, we compare and discuss outcomes from applying different search strategies to identify primary studies in a published SLR, an SLR update, and two replications in the area of effort estimation. These guidelines are then evaluated using an SLR in the area of software ecosystems, its update and a replication. The use of a single iteration forward snowballing with Google Scholar, and employing as a seed set the original SLR and its primary studies is the most cost-effective way to search for new evidence when updating SLRs. Furthermore, the importance of having more than one researcher involved in the selection of papers when applying the inclusion and exclusion criteria is highlighted through the results. Our proposed guidelines formulated based upon an effort estimation SLR, its update and two replications, were supported when using an SLR in the area of software ecosystems, its update and a replication. Therefore, we put forward that our guidelines ought to be adopted for updating SLRs in SE.

[Context] Systematic Literature Reviews (SLRs) have been adopted within the Software Engineering (SE) domain for more than a decade to provide meaningful summaries of evidence on several topics. Many of these SLRs are now outdated, and there are no standard proposals on how to update SLRs in SE. [Objective] The goal of this paper is to provide recommendations on how to best to search for evidence when updating SLRs in SE. [Method] To achieve our goal, we compare and discuss outcomes from applying different search strategies to identifying primary studies in a previously published SLR update on effort estimation. [Results] The use of a single iteration forward snowballing with Google Scholar, and employing the original SLR and its primary studies as a seed set seems to be the most cost-effective way to search for new evidence when updating SLRs. [Conclusions] The recommendations can be used to support decisions on how to update SLRs in SE.

Context: Despite being defined during the first phase of the Systematic Literature Review (SLR) process, the protocol is usually refined when other phases are performed. Several researchers have reported their experiences in applying SLRs in Software Engineering (SE) however, there is still a lack of studies discussing the iterative nature of the protocol definition, especially how it should be perceived by researchers conducting SLRs. Objective: The main goal of this study is to perform a survey aiming to identify: (i) the perception of SE researchers related to protocol definition; (ii) the activities of the review process that typically lead to protocol refinements; and (iii) which protocol items are refined in those activities. Method: A survey was performed with 53 SE researchers. Results: Our results show that: (i) protocol definition and pilot test are the two activities that most lead to further protocol refinements; (ii) data extraction form is the most modified item. Besides that, this study confirmed the iterative nature of the protocol definition. Conclusions: An iterative pilot testcan facilitate refinements in the protocol.

Background: Several researchers have reported their experiences in applying secondary studies in Software Engineering (SE), however, there is a lack of studies discussing the distinction between Systematic Mappings (SMs) and Systematic Literature Reviews (SLRs).Aims: The objective of this paper is to present the results of a tertiary study conducted to collect and evaluate evidence to better understand similarities and differences between SLRs and SMs related to four aspects: research question, search string, search strategy and quality assessment.Method: We identified 170 secondary studies that were reviewed to answer a set of Research Questions (RQ) related to the practical conduction of secondary studies in SE.Results: Results show that both SLRs and SMs have generic RQs, broad search strings, and adopt automatic search as search strategy.However, quality assessment has been more widely adopted in SLRs.Conclusions: In practice, only the quality assessment is conducted differently in SLRs and SMs.

Context: In order to preserve the value of Systematic Literature Reviews (SLRs), they should be frequently updated including new studies produced after the conduction of the reviews.However, most of SLRs are outdated and there is a lack of works that support the conduction of SLRs updates.Objective: The main goal of this paper is to report our experience in updating two of our SLRs.Method: To update these two SLRs, we used automated techniques based on VTM (Visual Text Mining) to guarantee the presence of relevant studies.Results: From our experience, some factors to satisfactorily update SLRs were identified: (i) to adopt software tools to support the updating process; (ii) to provide as much as possible information of previous SLR; (iii) to involve researchers from previous SLR; and (iv) to reuse protocol from preliminar SLR.Conclusions: Reported lessons learned can be used as a basis of knowledge to guide researchers when updating their SLRs.

International audience

The benefits of using assessment tools for programming assignments have been widely discussed in computing education. However, as both researchers and instructors are unaware of the characteristics of existing tools, they are either not used or are reimplemented. This paper presents the results of a study conducted to collect and evaluate evidence about tools that assist in the assessment of programming assignments. To achieve our goal, we performed a systematic literature review since it provides an objective procedure for identifying the quantity of existing research related to a research question. The results identified subjects in the development of new assessment tools that researchers could better investigate and characteristics of assessment tools that could help instructors make selections for their programming courses.

Background: A Systematic Literature Review (SLR) is a methodology used to aggregate relevant evidence related to one or more research questions. Whenever new evidence is published after the completion of a SLR, this SLR should be updated in order to preserve its value. However, updating SLRs involves significant effort. Objective: The goal of this paper is to investigate the application of forward snowballing to support the update of SLRs. Method: We compare outcomes of an update achieved using the forward snowballing versus a published update using the search-based approach, i.e., searching for studies in electronic databases using a search string. Results: Forward snowballing showed a higher precision and a slightly lower recall. It reduced in more than five times the number of primary studies to filter however missed one relevant study. Conclusions: Due to its high precision, we believe that the use of forward snowballing considerably reduces the effort in updating SLRs in Software Engineering; however the risk of missing relevant papers should not be underrated.

One of the activities associated with the Systematic Literature Review (SLR) process is the selection review of primary studies. When the researcher faces large volumes of primary studies to be analyzed, the process used to select studies can be arduous. In a previous experiment, we conducted a pilot test to compare the performance and accuracy of PhD students in conducting the selection review activity manually and using Visual Text Mining (VTM) techniques. The goal of this paper is to describe a replication study involving PhD and Master students. The replication study uses the same experimental design and materials of the original experiment. This study also aims to investigate whether the researcher's level of experience with conducting SLRs and research in general impacts the outcome of the primary study selection step of the SLR process. The replication results have confirmed the outcomes of the original experiment, i.e., VTM is promising and can improve the performance of the selection review of primary studies. We also observed that both accuracy and performance increase in function of the researcher's experience level in conducting SLRs. The use of VTM can indeed be beneficial during the selection review activity.

Context: In order to preserve the value of Systematic Reviews (SRs), they should be frequently updated considering new evidence that has been produced since the completion of the previous version of the reviews. However, the update of an SR is a time consuming, manual task. Thus, many SRs have not been updated as they should be and, therefore, they are currently outdated. Objective: The main contribution of this paper is to support the update of SRs. Method: We propose USR-VTM, an approach based on Visual Text Mining (VTM) techniques, to support selection of new evidence in the form of primary studies. We then present a tool, named Revis, which supports our approach. Finally, we evaluate our approach through a comparison of outcomes achieved using USR-VTM versus the traditional (manual) approach. Results: Our results show that USR-VTM increases the number of studies correctly included compared to the traditional approach. Conclusions: USR-VTM effectively supports the update of SRs.

Background: A systematic literature review (SLR) is a methodology used to aggregate all relevant existing evidence to answer a research question of interest. Although crucial, the process used to select primary studies can be arduous, time consuming, and must often be conducted manually. Objective: We propose a novel approach, known as 'Systematic Literature Review based on Visual Text Mining' or simply SLR-VTM, to support the primary study selection activity using visual text mining (VTM) techniques. Method: We conducted a case study to compare the performance and effectiveness of four doctoral students in selecting primary studies manually and using the SLR-VTM approach. To enable the comparison, we also developed a VTM tool that implemented our approach. We hypothesized that students using SLR-VTM would present improved selection performance and effectiveness. Results: Our results show that incorporating VTM in the SLR study selection activity reduced the time spent in this activity and also increased the number of studies correctly included. Conclusions: Our pilot case study presents promising results suggesting that the use of VTM may indeed be beneficial during the study selection activity when performing an SLR.