Context: The Open Science (OS) movement promotes the value of making public the research artifacts (datasets, analysis scripts, guidelines, etc.) used during empirical studies. OS is widely known in areas such as Medicine or Biology, where the process of sharing research artifacts is subject to strict protocols. Unfortunately, in Software Engineering (SE), this process is carried out in a non-systematic way, resulting in incomplete or inaccurate material shared by researchers, which hinders the reproducibility and replicability of empirical studies. Nevertheless, in recent years, it seems that the Empirical Software Engineering (ESE) community is embracing some of the proposed OS initiatives, such as the one proposed by the Association for Computing Machinery (ACM), which provides a badge system to evaluate the quality of a research artifact. This badge system has been adopted by several SE conferences as a method of assessing research artifacts. Aims: Focusing on human-oriented experiments (HOEs) in SE, whose research artifacts are more complex than those for computational experiments, this work applies Design Science Research (DSR) with a twofold purpose: (i) review the current status of HOEs research artifacts publication through evaluation of this practice in the most relevant ESE journals , and (ii) propose a structured outline for HOEs research artifacts driven by the aforementioned ACM badging policy. Method: Regarding the first purpose, we carried out a survey to analyze the current status of the publication of research artifacts considering relevant peer review journals and the quality of 106 research artifacts published in these journals with respect to the ACM badging policy. For the second purpose, an iterative process was carried out to review the content of 106 research artifacts research and their concordance with ACM badges, obtaining a structured scheme for HOEs research artifacts that has been validated through a detailed review of 12 research artifacts obtained from some of those of ACM badges in relevant SE conferences. In addition, we validated the proposal in the research artifacts of 2 of our own experiments. Results: Our survey reveals issues such as the 39,70% of journal studies making completely accessible their research artifacts; most of the analyzed research artifacts are incomplete; the most common repositories used in the ESE community to share the research artifacts are GitHub, institutional repositories, and Zenodo. On the other hand, the validated and structured research artifact outline consists of a list of ordered sections containing a set of artifacts, which can be mandatory or not to achieve a particular ACM badge. For its internal validation, several improvement iterations on the first release of the outline have been carried out based on the conference guidelines, the ACM badging policy, and other relevant proposals. Conclusions: Although the ESE community is making great efforts in standardization, review, and digital publishing related to OS, the availability and completeness of research artifacts can be improved. Our proposal for the elaboration of structured research artifact outline meets the requirements of HOEs in SE. Nevertheless, further research is needed not only to improve and externally validate it but also to disseminate its use among the research community.

Abstract In the context of requirements specification in Global Software Development, aspects such as differences in culture, language and schedule affect software development teams; however, we do not know the effect of these issues. Compare the native language requirements with the foreign language requirements concerning external quality. We conducted a controlled experiment of one‐factor two treatments within‐subjects with 17 experimental subjects. Wilcoxon test indicates that there is evidence to reject the null hypothesis ( p ‐value = 0.008); there is a statistically significant difference. The external quality value obtained with native language requirements is superior to the external quality produced with the foreign language. The effect size equals an absolute value of 0.45, which corresponds to a medium effect. The language used in the requirements specification influences the external quality; using the native language in the requirements specification significantly increases the external quality. The result obtained in this research should be considered to evaluate the roles and English language skills of GSD team members and their effect on external software quality. We also suggest considering the English language skills of the experimental subjects in the experimental laboratories since language could influence the results of the experiments.