The use of gamified crowdsourcing mechanisms through serious games and games with a purpose (GWAPs) has emerged as an effective motivational strategy for enhancing performance in human intelligence tasks (HITs). In this systematic literature review, we examine the underlying characteristics of competitive and cooperative player-oriented GWAPs and how they can be leveraged to optimize crowdsourcing performance in completing batches of HITs. By exploring gamified crowdsourcing elements in GWAPs, we can evaluate the impact of these two types of player behaviors (i.e., competition and cooperation) on motivation and performance. We reviewed 27 publications and grouped them into five categories: player orientation, game elements and motivation, crowd work optimization, gamified knowledge collection, and comparative studies and best practices. Our research pinpoints the significance of intuitive task instructions, alignment of game elements with player motivations, and the role of competitive and cooperative dynamics in enhancing engagement and performance.

In recent years, the development of Decision Support Systems (DSS) has played an instrumental role in the advancement of offshore renewable energy projects, particularly within the blue energy sector. Notwithstanding the technological advancements that have been made, the acceleration of such projects continues to be impeded by significant obstacles related to stakeholder engagement, feasibility assessment, and policy compliance. The objective of this study is to propose a design for a DSS for accelerating the construction of blue offshore energy platforms. This is to address the aforementioned challenges by integrating insights from stakeholder feedback and innovation trends. A participatory action study was conducted through a workshop with a diverse group of experts (n=20), including policymakers, practitioners, researchers, and public entities involved in offshore energy projects. The evaluation facilitated the determination of the DSS’ s efficacy in addressing user requirements and the identification of areas for enhancement. This study proposes a model for integrating stakeholder insights into technological solutions for offshore energy installations, thus offers significant contributions to the domain of sustainable blue energy development.

Online reviews are a crucial asset for e-commerce platforms as they provide consumers with valuable insights into products. It is important to note that these reviews are subjective and may contain biases. Therefore, it is essential to approach them with a critical eye. Despite this, online reviews remain a valuable tool for consumers when making purchasing decisions. This study focuses on developing web-based mini-games that target cognitive biases. The games are specifically designed to enhance the perception of e-commerce online reviews. A pilot study involving 85 participants was conducted to explore the potential of integrating these cognitive bias games into web platforms. The findings indicate promising avenues for leveraging these games to enhance cognitive personalization and improve the quality of e-commerce online reviews.

Large Language Models (LLMs) are widely used today in virtual assistants and content generation. However, there are suspicions that LLMs present confirmation bias, responding in a way that reinforces beliefs or assumptions embedded in users' questions, which can lead to erroneous decision-making, especially in sensitive areas such as healthcare. The objective of this research is to determine how often and under what conditions LLMs present confirmation bias and to identify the causes of this effect. The methodology involves conducting an experiment in which 52 biased healthcare questions are presented to 10 of the most popular models and analyzing whether their responses were biased. This work proves with statistical power the behavior of confirmation bias. We show that confirmation bias in LLMs occurs in all LLMs with a frequency of 20% to 60% of the occasions. The evidence suggests that the bias arises from the training database, the Transformer architecture itself, and the instructions in the fine-tuning phase by the companies behind the LLMs. This research explores pathways for the development of trustworthy LLMs.

Accessibility in digital labor is a research line that has been trending over the last few years. The usage of crowdsourcing, especially in the form of microtasks, can become an inclusive solution to support accessible digital work. Integrating cognitive abilities tests and task fingerprinting has proven to be effective mechanisms for microtask personalization when considering neurotypical people. In this article, we report the elaboration of usability tests on microtask personalization with neurodivergent people. The preliminary study recruited six participants with autism, attention deficit hyperactivity disorder, and dyslexia. The results obtained indicate that this solution can be inclusive and increase the accessibility of crowdsourcing tasks and platforms. One limitation of this study is that it is essential to evaluate this solution on a large scale to ensure the identification of errors and/or features of cognitive personalization in microtask crowdsourcing.

Abstract Online microtask labor has increased its role in the last few years and has provided the possibility of people who were usually excluded from the labor market to work anytime and without geographical barriers. While this brings new opportunities for people to work remotely, it can also pose challenges regarding the difficulty of assigning tasks to workers according to their abilities. To this end, cognitive personalization can be used to assess the cognitive profile of each worker and subsequently match those workers to the most appropriate type of work that is available on the digital labor market. In this regard, we believe that the time is ripe for a review of the current state of research on cognitive personalization for digital labor. The present study was conducted by following the recommended guidelines for the software engineering domain through a systematic literature review that led to the analysis of 20 primary studies published from 2010 to 2020. The results report the application of several cognition theories derived from the field of psychology, which in turn revealed an apparent presence of studies indicating accurate levels of cognitive personalization in digital labor in addition to a potential increase in the worker’s performance, most frequently investigated in crowdsourcing settings. In view of this, the present essay seeks to contribute to the identification of several gaps and opportunities for future research in order to enhance the personalization of online labor, which has the potential of increasing both worker motivation and the quality of digital work.

Determining the relatedness of publications by detecting similarities and connections between researchers and their outputs can help science stakeholders worldwide to find areas of common interest and potential collaboration. To this end, many studies have tried to explore authorship attribution and research similarity detection through the use of automatic approaches. Nonetheless, inferring author research relatedness from imperfect data containing errors and multiple references to the same entities is a long-standing challenge. In a previous study, we conducted an experiment where a homogeneous crowd of volunteers contributed to a set of author name disambiguation tasks. The results demonstrated an overall accuracy higher than 75% and we also found important effects tied to the confidence level indicated by participants in correct answers. However, this study left many open questions regarding the comparative accuracy of a large heterogeneous crowd with monetary rewards involved. This paper seeks to address some of these unanswered questions by repeating the experiment with a crowd of 140 online paid workers recruited via MTurk’s microtask crowdsourcing platform. Our replication study shows high accuracy for name disambiguation tasks based on authorship-level information and content features. These findings can be of greater informative value since they also explore hints of crowd behavior activity in terms of time duration and mean proportion of clicks per worker with implications for interface and interaction design.

When crowd workers provide their contributions in a shared working environment, they may be influenced by the inputs of other contributors in implicit ways. Stigmergy in crowdsourcing consists of tracking changes in work activities to guide crowd workers based on the digital traces left by other workers. In such scenarios, there is no direct communication between the contributors. Still, the traceable changes they left during their actions act as a mediating element that clearly affects the final work product. From a behavior analysis perspective, the properties recorded in event logs can be of practical value in observing the behavioral traces produced by crowd workers when performing microtasks. This form of task fingerprinting has been explored for over a decade to better understand performance-related data and user navigational behavior in crowdsourcing markets. In line with this, the goal of this paper is to study the feasibility of task fingerprinting alongside the stigmergic effect occurring in a crowdsourcing setting through a user event logger. To this end, a case study was conducted using a real-world scenario of extreme weather phenomena represented on interactive maps. Each user could observe the traces of other crowd members while providing annotations. Twelve experts in weather forecasting were recruited to participate in this study to annotate extreme weather events. The results indicate that it is possible to use task fingerprinting for tracking the stigmergic effect in such activities with gains in terms of implicit coordination. Furthermore, the task fingerprinting allowed to map participants with similar behavioral traces, suggesting an increase in the accuracy of annotation clusters.

Visualizing and examining the intellectual landscape and evolution of scientific communities to support collaboration is crucial for multiple research purposes. In some cases, measuring similarities and matching patterns between research publication document sets can help to identify people with similar interests for building research collaboration networks and university–industry linkages. The premise of this work is assessing feasibility for resolving ambiguous cases in similarity detection to determine authorship with natural language processing (NLP) techniques so that crowdsourcing is applied only in instances that require human judgment. Using an NLP-crowdsourcing convergence strategy, we can reduce the costs of microtask crowdsourcing while saving time and maintaining disambiguation accuracy over large datasets. This article contributes a next-gen crowd-artificial intelligence framework that used an ensemble of term frequency-inverse document frequency and bidirectional encoder representation from transformers to obtain similarity rankings for pairs of scientific documents. A sequence of content-based similarity tasks was created using a crowd-powered interface for solving disambiguation problems. Our experimental results suggest that an adaptive NLP-crowdsourcing hybrid framework has advantages for inter-researcher similarity detection tasks where fully automatic algorithms provide unsatisfactory results, with the goal of helping researchers discover potential collaborators using data-driven approaches.

Crowdsourcing has received considerable attention over the last fifteen years and has been the subject of several experiments that demonstrate its large potential for use in real-world situations. With the rapid growth of and access to crowd work environments, there is a need for new ways to ensure more equitable access for all people. Task design is one of the core aspects of the crowdsourcing process and its optimization is a priority for many requesters that want to have their tasks solved in short times and with high levels of accuracy. Aligned with this goal, a cognitive personalization framework can make it feasible to assess the information processing preferences of crowd workers in order to provide a useful user interface (UI) adaptation. In an effort to address this issue, this study recruited a total of 64 crowd workers to take cognitive style tests and perform prototypical tasks. The results indicate that it is possible to apply short tests and then obtain some useful indicators for better matching tasks to workers with implications for improving the general outcomes and acceptance rates in crowdsourcing.

Despite decades of research and development in named entity resolution, dealing with name ambiguity is still a challenging issue for many bibliometric-enhanced information retrieval (IR) tasks. As new bibliographic datasets are created as a result of the upward growth of publication records worldwide, more problems arise when considering the effects of errors resulting from missing data fields, duplicate entities, misspellings, extra characters, etc. As these concerns tend to be of large-scale, both the general consistency and the quality of electronic data are largely affected. This paper presents an approach to handle these name ambiguity problems through the use of crowdsourcing as a complementary means to traditional unsupervised approaches. To this end, we present “AuthCrowd”, a crowdsourcing system with the ability to decompose named entity disambiguation and entity matching tasks. Experimental results on a real-world dataset of publicly available papers published in peer-reviewed venues demonstrate the potential of our proposed approach for improving author name disambiguation. The findings further highlight the importance of adopting hybrid crowd-algorithm collaboration strategies, especially for handling complexity and quantifying bias when working with large amounts of data.

It has been a quarter of a century since the publication of the first edition of the IEEE International Conference on Computer Supported Cooperative Work in Design (CSCWD) held in 1996 in Beijing, China. Despite some attempts to empirically examine the evolution and identity of the field of CSCW and its related communities and disciplines, the scarcity of scientometric studies on the IEEE CSCWD research productivity is noteworthy. To fill this gap, this study reports on an exploratory quantitative analysis of the literature published in the IEEE CSCWD conference proceedings with the purpose of visualizing and understanding its structure and evolution for the 2001-2019 period. The findings offer valuable insights into the paper and author distribution, country and citation-level productivity indicators, degree of collaboration, and collaboration index. Through this analysis we also expect to get an initial overview of the IEEE CSCWD conference concerning the main topics being presented, most cited papers, and variances in the number of keywords, full-text views, and references.