The use of gamified crowdsourcing mechanisms through serious games and games with a purpose (GWAPs) has emerged as an effective motivational strategy for enhancing performance in human intelligence tasks (HITs). In this systematic literature review, we examine the underlying characteristics of competitive and cooperative player-oriented GWAPs and how they can be leveraged to optimize crowdsourcing performance in completing batches of HITs. By exploring gamified crowdsourcing elements in GWAPs, we can evaluate the impact of these two types of player behaviors (i.e., competition and cooperation) on motivation and performance. We reviewed 27 publications and grouped them into five categories: player orientation, game elements and motivation, crowd work optimization, gamified knowledge collection, and comparative studies and best practices. Our research pinpoints the significance of intuitive task instructions, alignment of game elements with player motivations, and the role of competitive and cooperative dynamics in enhancing engagement and performance.

Scientometrics is undergoing a methodological transformation driven by the increasing availability of large-scale scientific data and advances in artificial intelligence (AI). Traditional approaches centered on citation analysis and bibliographic coupling are now complemented by methods that leverage semantic representations, structured knowledge, and natural language understanding. However, current generative AI systems pose inherent challenges such as hallucinations, lack of transparency in decision-making and explainability, and issues with source reliability. In this article, we seek to mitigate these challenges by outlining a framework that integrates knowledge graphs (KGs) and large language models (LLMs) for scientometric inquiry. Taking a socio-technical perspective, the article sets out to explore the intersectional space of computing and information science from a human-centered approach. The framework is designed to support both established scientometric tasks, such as trend analysis, topic detection, and collaboration mapping, and more exploratory, insight-generating applications, including question answering, knowledge discovery, and contextual enrichment of scientific content. Drawing on recent developments in the use of KGs and LLMs in scientific domains, we provide a comparative overview of existing work, identify key design principles, and discuss the advantages and limitations of such a framework. Our goal is to chart a pathway toward more interactive, transparent, and generative approaches in information science and cross-disciplinary research. • Multidimensional mapping from large bibliographic datasets remains challenging. • A KG-LLM framework is proposed to support interactive scientometric inquiry. • HITL validation addresses the black-box limitations of KG- and LLM-enabled solutions. • Implications for knowledge-based question answering are derived.

In this paper, we explore a perspective on well-being in the context of digital nomads, reporting the first findings of an empirical study investigating the extent to which the nomadic lifestyle enables a healthy life and the enjoyment of well-being. Following the Gioia methodology, we collected and analyzed textual data from posts in a specific online Reddit social network discussion community to do this. Preliminary findings indicate that Fear of Missing Out (FOMO) acts as a trigger that stimulates the adoption of digital nomad lifestyles.

The polarization of ideas is a natural phenomenon of human behavior. When productive, it can generate scenarios for reflection on arguments, opinions, and interactions, but when polarization is extreme, it generates social isolation, manipulation, perception of injustice, unproductive debates, certainties of absolute truths, fanaticism, feelings of hatred, and even wars. Deconstructing the ecosystem of extreme polarization directly benefits social stability. Virtual social environments are excellent scenarios for fostering debates, negotiations, and the exposition of ideas. The objective of this study is to use the same virtual social ecosystem with its empowerment mechanisms and influence attributes to support the creation of tools, functionalities, and simulations to develop healthier systems. The knowledge generated supported the development of a design and cognitive ethical affordances practiced in metacognitive design, optimized navigation, and metacognitive simulations. The cognitive, ethical affordances incorporated into the Productive Reinforcement Spiral method model were part of a metacognitive simulation of collaborative negotiation with a productive reinforcement spiral mechanism, whose description and evaluation of the design are the main objectives of this paper.

This Participatory Action Design Research (PADR) study explores how persuasive technologies and the Internet of Things (IoT) can be used to encourage participation in recycling activities. By involving primary school students in the design process of a sociotechnical solution through participatory methods, we aim to create intelligent and persuasive applications that effectively promote sustainable behaviors in the school environment. To address the objectives of this investigation, the PADR methodology was adapted to the context of persuasive technologies by incorporating some critical activities, such as defining target behaviors. We argue that applying a participatory action design research approach involving users in all the stages of the design process of persuasive technologies for sustainability can lead to engaging and intuitive solutions tailored to the specific needs of users in real-world contexts.

Collaborative crowdsourcing has evolved over recent years as a problem-solving model that builds on interactions that take place within crowds or self-organizing teams that form and dissolve in response to an open call. Unlike microtasking, its effectiveness depends primarily on collective actions rather than individual efforts, particularly for non-decomposable tasks that require a certain level of interdependency. Although social news curation has gained attention as a means of empowering both citizens and journalists, there is a need to understand the components and constraints that shape this online crowdsourcing activity when mediated by artificial intelligence (AI). In response to this gap, this paper reimagines the traditional social news curation model through a design fiction prototype consisting of a personalized AI system and user-driven social curation activities. In a highly polarized opinion and political climate where propaganda, misinformation, and extreme polarization take on a dominant framing to generate social chaos, the envisioned solution seeks to model healthy and trustworthy socio-algorithmic interactions towards a more democratic and relevant form of news curation.

The rise of artificial intelligence agents and information technology platforms governed by data-driven algorithms has had profound impacts on the way knowledge work is managed and carried out. Concerning the novel uses of digital work platforms, the arising tensions might suggest that the socio-technical design frameworks adopted by major tech firms are becoming less viable and in some cases ineffective. We propose a conceptual architecture to support a socio-technical framework for exploring Personal Knowledge Ecologies in the context of the digital knowledge economy. This framework aims to increase the degree of individual freedom for the interoperation of these knowledge information infrastructures.

Accessibility in digital labor is a research line that has been trending over the last few years. The usage of crowdsourcing, especially in the form of microtasks, can become an inclusive solution to support accessible digital work. Integrating cognitive abilities tests and task fingerprinting has proven to be effective mechanisms for microtask personalization when considering neurotypical people. In this article, we report the elaboration of usability tests on microtask personalization with neurodivergent people. The preliminary study recruited six participants with autism, attention deficit hyperactivity disorder, and dyslexia. The results obtained indicate that this solution can be inclusive and increase the accessibility of crowdsourcing tasks and platforms. One limitation of this study is that it is essential to evaluate this solution on a large scale to ensure the identification of errors and/or features of cognitive personalization in microtask crowdsourcing.

Abstract Online microtask labor has increased its role in the last few years and has provided the possibility of people who were usually excluded from the labor market to work anytime and without geographical barriers. While this brings new opportunities for people to work remotely, it can also pose challenges regarding the difficulty of assigning tasks to workers according to their abilities. To this end, cognitive personalization can be used to assess the cognitive profile of each worker and subsequently match those workers to the most appropriate type of work that is available on the digital labor market. In this regard, we believe that the time is ripe for a review of the current state of research on cognitive personalization for digital labor. The present study was conducted by following the recommended guidelines for the software engineering domain through a systematic literature review that led to the analysis of 20 primary studies published from 2010 to 2020. The results report the application of several cognition theories derived from the field of psychology, which in turn revealed an apparent presence of studies indicating accurate levels of cognitive personalization in digital labor in addition to a potential increase in the worker’s performance, most frequently investigated in crowdsourcing settings. In view of this, the present essay seeks to contribute to the identification of several gaps and opportunities for future research in order to enhance the personalization of online labor, which has the potential of increasing both worker motivation and the quality of digital work.

The Massification of remote work, in response to the COVID-19 pandemic, has been causing significant changes in productive and working arrangements, both for individuals, organizations, and society. At the level of personal life philosophy, for example, this transformation can be evidenced in the dissemination of digital nomadism values such as work/leisure balance among corporate workers. On the other hand, at the level of gig/crowd work platforms, the emergence of tensions may indicate the exhaustion of sociotechnical design models adopted by big techs. We show how evidence collected from digital nomads in empirical ethnography studies can inform HCI/CSCWD researchers of design-oriented strands to explore emerging opportunities in new creative digital crypto-economic ecosystems. Finally, we present a proposed research agenda to explore DNs’ activities in the crypto-economic ecosystem.

This paper explores the challenges and strategies used to design, build and evaluate a collective intelligence (CI) model to support discussions about cities. Through an autoethnography of ideas and discussion practices in both online and offline contexts, this work explores the tensions between the chosen methodological approaches, including design science research (DSR) and participatory action research (PAR). Moreover, we also examine the challenges and pitfalls observed during the practical conduction of this research when involving participants in the process of empirically evaluating the proposed model. Finally, aspects related to the autoethnographic process itself as a reflective method are discussed alongside the consequences of desiring and seeking participation in the research process.

Determining the relatedness of publications by detecting similarities and connections between researchers and their outputs can help science stakeholders worldwide to find areas of common interest and potential collaboration. To this end, many studies have tried to explore authorship attribution and research similarity detection through the use of automatic approaches. Nonetheless, inferring author research relatedness from imperfect data containing errors and multiple references to the same entities is a long-standing challenge. In a previous study, we conducted an experiment where a homogeneous crowd of volunteers contributed to a set of author name disambiguation tasks. The results demonstrated an overall accuracy higher than 75% and we also found important effects tied to the confidence level indicated by participants in correct answers. However, this study left many open questions regarding the comparative accuracy of a large heterogeneous crowd with monetary rewards involved. This paper seeks to address some of these unanswered questions by repeating the experiment with a crowd of 140 online paid workers recruited via MTurk’s microtask crowdsourcing platform. Our replication study shows high accuracy for name disambiguation tasks based on authorship-level information and content features. These findings can be of greater informative value since they also explore hints of crowd behavior activity in terms of time duration and mean proportion of clicks per worker with implications for interface and interaction design.

When crowd workers provide their contributions in a shared working environment, they may be influenced by the inputs of other contributors in implicit ways. Stigmergy in crowdsourcing consists of tracking changes in work activities to guide crowd workers based on the digital traces left by other workers. In such scenarios, there is no direct communication between the contributors. Still, the traceable changes they left during their actions act as a mediating element that clearly affects the final work product. From a behavior analysis perspective, the properties recorded in event logs can be of practical value in observing the behavioral traces produced by crowd workers when performing microtasks. This form of task fingerprinting has been explored for over a decade to better understand performance-related data and user navigational behavior in crowdsourcing markets. In line with this, the goal of this paper is to study the feasibility of task fingerprinting alongside the stigmergic effect occurring in a crowdsourcing setting through a user event logger. To this end, a case study was conducted using a real-world scenario of extreme weather phenomena represented on interactive maps. Each user could observe the traces of other crowd members while providing annotations. Twelve experts in weather forecasting were recruited to participate in this study to annotate extreme weather events. The results indicate that it is possible to use task fingerprinting for tracking the stigmergic effect in such activities with gains in terms of implicit coordination. Furthermore, the task fingerprinting allowed to map participants with similar behavioral traces, suggesting an increase in the accuracy of annotation clusters.

This paper proposes an agenda of possible solutions relevant to the challenge of polarization of extreme positions. The challenge of mitigating polarization of extreme positions has been a hot topic in politics, journalism, new media, and, more recently, in CSCW and HCI research. In this study, the main focus is to investigate the harmful effects of polarization from a neuroscience point of view. The suggested solutions are based on theories of social cognitive neuroscience and encourage awareness of inappropriate behavior, an increase in the diversity of arguments, and the exercise of individual and collective responsibility, to mitigate the problem of polarization that threatens public debate and, consequently, democracy.

In this scoping review, we aim to summarize and analyze the latest persuasive design research developments for smart bins. This study initially collected data from 551 scientific papers, and later, based on a selection process, 13 papers that mainly focused on handling smart bins through persuasive designs were included in the final analysis. These 13 papers were rated by four characteristics: research specifications, methodologies, persuasive strategies, and adopted technologies. We argue that, to understand how to design cyber-physical systems for waste management that involve the cooperation of civil society, a promising path includes unraveling how persuasive smart bins designs are being developed and identifying the challenges and opportunities that exist for waste management in cyber-physical collaborative environments, at the intersection of person, place, and technology.

In this paper, we continue our investigations on digital nomadism and the impact of COVID-19 pandemic on the work-related aspects and lifestyle of digital nomads (DN). The findings presented in this empirical study reflect the analysis of the impact of COVID-19 outbreak (and its waves) on the market economy and work-life boundaries of DNs as perceived from posts and comments gathered from a Reddit community during the period of early March 2020 until the end of 2022. From this point, our results indicate that the massification of remote work among formal workers in response to COVID-19 pandemic has impacted both the formal labor market and the DN ecosystem. As a consequence, we argue that digital nomadism tends to play a critical role beyond work from (almost) anywhere (WFA) in a post-COVID-19 era taking into account the novel facets of nomadic work-lifestyle.

Visualizing and examining the intellectual landscape and evolution of scientific communities to support collaboration is crucial for multiple research purposes. In some cases, measuring similarities and matching patterns between research publication document sets can help to identify people with similar interests for building research collaboration networks and university–industry linkages. The premise of this work is assessing feasibility for resolving ambiguous cases in similarity detection to determine authorship with natural language processing (NLP) techniques so that crowdsourcing is applied only in instances that require human judgment. Using an NLP-crowdsourcing convergence strategy, we can reduce the costs of microtask crowdsourcing while saving time and maintaining disambiguation accuracy over large datasets. This article contributes a next-gen crowd-artificial intelligence framework that used an ensemble of term frequency-inverse document frequency and bidirectional encoder representation from transformers to obtain similarity rankings for pairs of scientific documents. A sequence of content-based similarity tasks was created using a crowd-powered interface for solving disambiguation problems. Our experimental results suggest that an adaptive NLP-crowdsourcing hybrid framework has advantages for inter-researcher similarity detection tasks where fully automatic algorithms provide unsatisfactory results, with the goal of helping researchers discover potential collaborators using data-driven approaches.

In this paper we present findings from a bibliometric evaluation of scientific publications on human-AI systems, indexed in the Dimensions database over the past five years (2018 to 2022). The study maps the research landscape in this burgeoning area, as it relates to the topic of collaboration. To this end, we assessed publication and citation counts over time, authorship-level indicators, and keyword occurrence frequency. We also examined funding information as an indicator of research priorities, alongside usage-based statistics and alternative metrics such as social media mentions, recommendations, and reads. Our preliminary findings highlight a significant focus on aspects like trust, explainability, transparency, and autonomy in highly complex scenarios through the use of generative models and hybrid interaction techniques. The results also reveal a growth in the number of publications and funding grants, although a certain lack of maturity is observable in terms of citation patterns and coherence of thematic clusters.

In this paper, we report on new findings about the results of an empirical study which aims to investigate how the COVID-19 pandemic has been shaping nomadic work practices and also challenging the lifestyles of digital nomads (DN). To do this, we collected textual data from posts in a Reddit community. We argue that, in order to understand how to design technical solutions for the so-called ‘new normal’ working conditions, one way to approach this is to understand how digital nomads are being impacted in their work practices and routines, and also how they are seeing the future of their technology-mediated work-life space. Finally, we show how evidence collected from DNs about their experiences and difficulties perceived during the pandemic period can inform CSCW researchers worldwide about future design-oriented strands.

In recent years, digital technologies have been used to support discussions about the city and also to involve citizens in participatory public processes. However, despite the widespread use of social media platforms, old issues related to engagement and participation still persist in digital initiatives. The main goal of this study is to carry out an empirical evaluation of a collective intelligence model that combines crowdsourcing and social storytelling to support discussions about the city from a bottom-up perspective. Within a design science research approach we designed a participatory action study that was carried out through a workshop with students and professionals from different areas, such as architecture, urban design and information technology. As a result, we were able to assess whether the collective intelligence model was acceptable to the participants by investigating whether the behavioral assumptions were valid and thus outlining some contributions to the field of urban informatics.

Crowdsourcing has received considerable attention over the last fifteen years and has been the subject of several experiments that demonstrate its large potential for use in real-world situations. With the rapid growth of and access to crowd work environments, there is a need for new ways to ensure more equitable access for all people. Task design is one of the core aspects of the crowdsourcing process and its optimization is a priority for many requesters that want to have their tasks solved in short times and with high levels of accuracy. Aligned with this goal, a cognitive personalization framework can make it feasible to assess the information processing preferences of crowd workers in order to provide a useful user interface (UI) adaptation. In an effort to address this issue, this study recruited a total of 64 crowd workers to take cognitive style tests and perform prototypical tasks. The results indicate that it is possible to apply short tests and then obtain some useful indicators for better matching tasks to workers with implications for improving the general outcomes and acceptance rates in crowdsourcing.

The purpose of this paper is to reveal a set of new results obtained from an ongoing investigation focused on the way that the particular characteristics which are inherent to “wannabe” digital nomads’ activities contribute to the sustainability of the whole digital nomad ecosystem. In line with the premise of this research, we assume the importance of understanding the impacts that are being felt in the personal knowledge management ecology practices and routines of digital nomads as experienced by a specific online population (i.e., Reddit user base), together with a deep and wide examination of their preferences and expectations regarding the technology-mediated work-life issues that exert a direct influence on the digital nomad community. To this end, we gathered and further processed text posts and comments from users in the ‘/r/digitalnomad’ subreddit. From a sociotechnical standpoint, the empirical data extracted from this sample population about the wannabe/how to be digital nomad symbiotic ecosystem can provide insightful information for researchers worldwide about future design-level interventions.

With the increasing development of human-AI teaming structures within and across geographies, the time is ripe for a continuous and objective look at the predictors, barriers, and facilitators of human-AI scientific collaboration from a multidisciplinary point of view. This paper aims at contributing to this end by exploiting a set of factors affecting attitudes towards the adoption of human-AI interaction into scientific work settings. In particular, we are interested in identifying the determinants of trust and acceptability when considering the combination of hybrid human-AI approaches for improving research practices. This includes the way as researchers assume human-centered artificial intelligence (AI) and crowdsourcing as valid mechanisms for aiding their tasks. Through the lens of a unified theory of acceptance and use of technology (UTAUT) combined with an extended technology acceptance model (TAM), we pursue insights on the perceived usefulness, potential blockers, and adoption drivers that may be representative of the intention to use hybrid intelligence systems as a way of unveiling unknown patterns from large amounts of data and thus enabling novel scientific discoveries.

We report the first findings of an empirical study aimed at investigating how COVID-19 pandemic has impacted the work practices and lifestyles of digital nomads (DN). To do this, we analyzed messages, questions and comments posted by digital nomads in a specific online discussion community of the Reddit social network. Preliminary findings indicate COVID-19 as an opportunity to test DN lifestyle by aspiring digital nomads who want to plan their careers and also present evidence of an overload of online channels for actual DNs. On the other hand, we found that much of the literature on digital nomadism is fragmented and scattered through different disciplines and perspectives, with a strong focus on digital nomads' lifestyles. In order to obtain a holistic and unified understanding of digital nomads, we conducted a comprehensive literature review to further conceptualize the phenomenon under study.

Over the past decades, the use of digital technologies to support participatory urban planning and design has been repeatedly described as a crucial instrument and critical building block for tackling historical problems of participation in such processes. Social media, e-participation platforms, and crowdsourcing applications are examples of technologies that can involve citizens in decision-making processes and thus leverage the benefits of collective intelligence. However, despite the extensive use of social media platforms, old problems related to engagement and participation still occur in digital initiatives. Successful collaboration examples between citizens, policymakers, and strategic stakeholders are still scarce based on online social practices. This study aims to introduce a collective intelligence model, which combines crowdsourcing and social storytelling to support participatory urban planning and design from a bottom-up perspective. The paper concludes by discussing a scenario where citizens can engage in mapping, taking photos, sending ideas, or even creating collective stories about their university issues in a post-pandemic future.

Despite decades of research and development in named entity resolution, dealing with name ambiguity is still a challenging issue for many bibliometric-enhanced information retrieval (IR) tasks. As new bibliographic datasets are created as a result of the upward growth of publication records worldwide, more problems arise when considering the effects of errors resulting from missing data fields, duplicate entities, misspellings, extra characters, etc. As these concerns tend to be of large-scale, both the general consistency and the quality of electronic data are largely affected. This paper presents an approach to handle these name ambiguity problems through the use of crowdsourcing as a complementary means to traditional unsupervised approaches. To this end, we present “AuthCrowd”, a crowdsourcing system with the ability to decompose named entity disambiguation and entity matching tasks. Experimental results on a real-world dataset of publicly available papers published in peer-reviewed venues demonstrate the potential of our proposed approach for improving author name disambiguation. The findings further highlight the importance of adopting hybrid crowd-algorithm collaboration strategies, especially for handling complexity and quantifying bias when working with large amounts of data.

It has been a quarter of a century since the publication of the first edition of the IEEE International Conference on Computer Supported Cooperative Work in Design (CSCWD) held in 1996 in Beijing, China. Despite some attempts to empirically examine the evolution and identity of the field of CSCW and its related communities and disciplines, the scarcity of scientometric studies on the IEEE CSCWD research productivity is noteworthy. To fill this gap, this study reports on an exploratory quantitative analysis of the literature published in the IEEE CSCWD conference proceedings with the purpose of visualizing and understanding its structure and evolution for the 2001-2019 period. The findings offer valuable insights into the paper and author distribution, country and citation-level productivity indicators, degree of collaboration, and collaboration index. Through this analysis we also expect to get an initial overview of the IEEE CSCWD conference concerning the main topics being presented, most cited papers, and variances in the number of keywords, full-text views, and references.

Citizen engagement in building user-curated narratives of complex or long-lasting news stories has been the key foundation of the design and implementation of the Acropolis virtual environment. Previous user studies have shown, by positive evidence, that this goal can be pragmatically achieved, but the challenge now lies in assessing: a) the extent to which an environment like Acropolis can be used to empower citizens; and b) whether and how the tool could be used to support the work of professional curators. Findings from a focus group study highlighted the tool's potential to engage citizens with news, the usefulness of the environment to build virtual memories, and the convenience of using Acropolis to support professional journalistic work.

With cutting edge scientific breakthroughs, human-centred algorithmic approaches have proliferated in recent years and information technology (IT) has begun to redesign socio-technical systems in the context of human-AI collaboration. As a result, distinct forms of interaction have emerged in tandem with the proliferation of infrastructures aiding interdisciplinary work practices and research teams. Concomitantly, large volumes of heterogeneous datasets are produced and consumed at a rapid pace across many scientific domains. This results in difficulties in the reliable analysis of scientific production since current tools and algorithms are not necessarily able to provide acceptable levels of accuracy when analyzing the content and impact of publication records from large continuous scientific data streams. On the other hand, humans cannot consider all the information available and may be adversely influenced by extraneous factors. Using this rationale, we propose an initial design of a human-AI enabled pipeline for performing scientometric analyses that exploits the intersection between human behavior and machine intelligence. The contribution is a model for incorporating central principles of human-machine symbiosis (HMS) into scientometric workflows, demonstrating how hybrid intelligence systems can drive and encapsulate the future of research evaluation.

Over the past decade, online crowdsourcing has established itself as an emerging paradigm that industry and government have been using to harness the cognitive abilities of a multitude of users distributed around the world. In this context, microtask crowdsourcing has become the method of choice for addressing a wide range of diverse problems. Microtasks typically require a minimum of time and cognitive effort, but combined individual efforts have made it possible to accomplish great achievements. The goal of this paper is to contribute to the ongoing effort of understanding whether the same success that microtask crowdsourcing has achieved in other domains can be obtained in the field of social news curation. In particular, we ask whether it is possible to turn online news curation, typically a social and collaborative activity on the Web, into a model in which curatorial activities are mapped into microtasks to be performed by a crowd of online users.

Recently, crowdsourcing platforms have been used to solve problems in the field of urban planning by involving crowds of citizens in performing tasks. However, the success of this approach is directly related to how work is managed. The goal of the present study is to make a broad characterization of work management in crowdsourcing approaches applied to urban planning through a systematic literature review. More specifically, we aim to investigate aspects related to the quality of work.

The structure and evolution of a scientific research community can be quantitatively assessed taking into account the interactions between scientific agents dispersed geographically. In the recent years, CSCW has stabilized as a cross-disciplinary field suffering significant changes in its core structure, and there is limited understanding about the factors influencing the nature and progress of collaborative computing research. In this paper, we measure the correlation between a set of features related to the influence of collaboration types on the number of citations as well as the geographical distribution of the accumulated contribution to the CSCW literature. Overall, our work can represent a starting point to demonstrate how the study of scientific collaboration can partly explain the variations in the number of citations, frequency of papers, and topics addressed.

In recent years, social curation technology has been successfully promoted as a means of addressing the problem of involving citizens with news content. Sites like Facebook., Reddit and Storify., just to name a few, allow their users to discuss, comment., share, identify and review news content from various sources. A key goal of this paper is to analyze the process of social validation in the context of Acropolis, a social computing platform that allows citizens to build and share their own narratives about complex or long-term news stories. The sense of social validation determines an increase in self-esteem, self-confidence, and instant happiness, when conveniently engendered in virtual environments.

A substantial amount of work is often overlooked due to the exponential rate of growth in global scientific output across all disciplines. Current approaches for addressing this issue are usually limited in scope and often restrict the possibility of obtaining multidisciplinary views in practice. To tackle this problem, researchers can now leverage an ecosystem of citizens, volunteers and crowd workers to perform complex tasks that are either difficult for humans and machines to solve alone. Motivated by the idea that human crowds and computer algorithms have complementary strengths, we present an approach where the machine will learn from crowd behavior in an iterative way. This approach is embodied in the architecture of SciCrowd, a crowd-powered human-machine hybrid system designed to improve the analysis and processing of large amounts of publication records. To validate the proposal's feasibility, a prototype was developed and an initial evaluation was conducted to measure its robustness and reliability. We conclude this paper with a set of implications for design.

Crowdsourcing has shown to be a valuable problem-solving approach to handle the increasing complexity and scale of tasks for which the current AI algorithms are still struggling. Crowd intelligence can be particularly useful to train and supervise AI systems in a symbiotic, co-evolutionary relationship that raises long-term research challenges to the hybrid, crowd-computing design space. With the increase in the scale of mixed-initiative approaches, we need to gain a better understanding of the implications of crowd-powered systems as a scaffold for AI through the study of massive crowd-machine interactions. In this paper, we identify some open challenges and design implications for future crowd-AI hybrid systems. A framework is also proposed based on the practical challenges of addressing human-centered AI methods and processes.