Crowd-based Requirements Engineering (CrowdRE) promotes the active involvement of a large number of stakeholders in RE activities. A prominent strand of CrowdRE research concerns the creation and use of online platforms for a crowd of stakeholders to formulate ideas, which serve as an additional input for requirements elicitation. Most of the reported case studies are of small size, and they analyze the size of the crowd, rather than the quality of the collected ideas. By means of an iterative design that includes three case studies conducted at two organizations, we present the CREUS method for crowd-based elicitation via user stories. Besides reporting the details of these case studies and quantitative results on the number of participants, ideas, votes, etc., a key contribution of this paper is a qualitative analysis of the elicited ideas. To analyze the quality of the user stories, we apply criteria from the Quality User Story framework, we calculate automated text readability metrics, and we check for the presence of vague words. We also study whether the user stories can be linked to software qualities, and the specificity of the ideas. Based on the results, we distill six key findings regarding CREUS and, more generally, for CrowdRE via pull feedback.

Crowd-based Requirements Engineering, or CrowdRE, is a recent paradigm that promotes the active participation of a large number of stakeholders in RE. In CrowdRE, requirements elicitation can be crowdsourced by creating an online platform that allows stakeholders to formulate ideas regarding the specific product. Although some case studies on crowd-based elicitation exist, no conclusive evidence can be derived on the effectiveness of such techniques. In this paper, we study crowd-based elicitation within a large governmental organization. We conduct two case studies of CrowdRE within the Royal Netherlands Marechaussee. For this, we construct the KMar-Crowd method, which adapts CrowdRE ideas to the needs of governmental organizations. While one case compares the crowd-generated ideas with requirements elicited via interviews and similar techniques, the other measures the usefulness of the gathered ideas for a product for which no prior elicitation was conducted. The results of the case studies, which attracted larger crowds than in previous studies, indicate that CrowdRE can be successfully applied to engage the users of a software product in requirements elicitation. Contradicting earlier studies, the inclusion of gamification elements in the CrowdRE method did not increase the motivation to participate, possibly due to the nature of governmental organizations.