Abstract As open project repositories have become widespread, evaluating the quality of open source software (OSS) has gained attention in the software community. Despite the variety in quality evaluation models or frameworks (QEMoF) proposed for OSS, their adoption in practice remains limited. To further investigate this argument, a systematic literature review (SLR) has been conducted to characterize the existing QEMoF for OSS, based on particular properties, and to examine comprehensively their content and structure for identifying the gap between theory and practice. In this context, a total of 36 primary studies in the period of 2003–2020 have been analyzed. It has been identified that the majority of QEMoF takes ISO/IEC 9126 as a reference, evaluates community‐based aspects, mostly evaluates maintainability and usability as quality attributes, supports subjective and quantitative evaluations, and faces challenges due to diverse and dynamic nature of OSS products. For a better overview of the state of the art in the field, evaluation criteria for comparing the overall quality of the QEMoF are identified and applied. Finally, overall findings are discussed, and suggestions are provided for practitioners and researchers to formulate more robust studies in the future, which can advance the current situation for the benefit of the software community.

As the number of available Open Source Software (OSS) and the interest they attract are increasing, numerous product attributes are provided to developers and users for evaluating the quality and success of an OSS. Accordingly, various articles in the literature assess the quality and success of OSS, by using different quality attributes and metrics and different approaches. Though this variety can be considered as a positive indicator of research interest and maturation on one side, it creates a kind of jungle in defining and understanding the terms 'quality' and 'success' on the other side. Based on this challenge, in this study, we targeted a systematic mapping (SM) of the articles on quality and success of OSS. More than 474 articles have appeared in this area between the years 2002 and 2017, and the final pool of 128 articles is obtained by defining and applying inclusion and exclusion criteria. SM was employed to develop a classification scheme and categorized the existing body of articles with respect to five research questions (RQs) on: contribution and research types, quality criteria and metrics, success criteria and metrics, the relation of quality and success, and demographics. We observed that the majority of the articles assess the concept of quality as 'code quality', whereas the concept of success is mostly perceived as 'market success' and/or 'developer activity'. Moreover, the metrics of 'contributing developers/users', and the quality attribute of 'functionality' are the quality criteria most employed in the assessment of success.