Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts’ ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers’ evaluation is, however, less reliable and might substantially differ from experts’ evaluation.

introduction Share on Second Workshop on Recommender Systems for Human Resources (RecSys in HR 2022) Authors: Toine Bogers Aalborg University Copenhagen, Denmark Aalborg University Copenhagen, DenmarkView Profile , David Graus Randstad Groep Nederland, Netherlands Randstad Groep Nederland, NetherlandsView Profile , Mesut Kaya Aalborg University Copenhagen, Denmark Aalborg University Copenhagen, DenmarkView Profile , Francisco Gutiérrez Computer Science, KU Leuven, Belgium Computer Science, KU Leuven, BelgiumView Profile , Sepideh Mesbah Randstad Groep Nederland, Netherlands Randstad Groep Nederland, NetherlandsView Profile , Chris Johnson Indeed, United States Indeed, United StatesView Profile Authors Info & Claims RecSys '22: Proceedings of the 16th ACM Conference on Recommender SystemsSeptember 2022Pages 671–674https://doi.org/10.1145/3523227.3547414Published:13 September 2022Publication History 0citation112DownloadsMetricsTotal Citations0Total Downloads112Last 12 Months42Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We introduce a Language-consistent multi-lingual Open Relation Extraction Model (LOREM) for finding relation tuples of any type between entities in unstructured texts. LOREM does not rely on language-specific knowledge or external NLP tools such as translators or PoS-taggers, and exploits information and structures that are consistent over different languages. This allows our model to be easily extended with only limited training efforts to new languages, but also provides a boost to performance for a given single language. An extensive evaluation performed on 5 languages shows that LOREM outperforms state-of-the-art mono-lingual and cross-lingual open relation extractors. Moreover, experiments on languages with no or only little training data indicate that LOREM generalizes to other languages than the languages that it is trained on.

Sepideh Mesbah, Jie Yang, Robert-Jan Sips, Manuel Valle Torre, Christoph Lofi, Alessandro Bozzon, Geert-Jan Houben. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

As crowdsourcing gains popularity, organisations seek ways to systematically and reliably involve their workforce with data processing pipelines. Mobile crowdsourcing allows for opportunistic task executions and thus, potentially, for higher throughput. However, how to engage and to retain employees in enterprise crowdsourcing campaigns is still an open research topic. This paper discusses the results of a study performed in IBM Benelux. We surveyed 93 employees to discover the factors that might affect engagement in mobile enterprise crowdsourcing. The survey informed the design of an experiment that aimed at investigating the effectiveness of different task notification strategies. We studied how factors such as time and context of notification can affect the participation and retention of employees. Results show that break times are the most suitable for crowd work, and that ``aggressive'' notification strategies act as deterrent for participation, while moderate yet regular nudges are the most likely to retain contributors.