AI systems are increasingly being positioned to assist people in decision-making. However, recent empirical studies show critical concerns that people over-rely on AI advice without analytically engaging with it. While HCI research explores how people rely on AI advice, we argue that it largely overlooks an important aspect: replicating realistic decision-making scenarios. Human-AI interaction factors influence people’s reliance on AI advice. To understand human-AI interaction factors and their interplay, we conducted an analytical review of recent studies in human-AI reliance literature. We analyzed the decision-making tasks in research and their validity in application-grounded contexts. Our findings show that user engagement is a precious commodity for relying on AI advice; however, it comes at a cost. We also discuss factors contributing to “appropriate reliance”, existing research gaps, and recommendations for intervention design for human-AI reliance. Our work contributes to the critical body of research on building appropriate reliance on AI advice.

We introduce BikeBench, an engineering design benchmark for evaluating generative models on problems with multiple real-world objectives and constraints. As generative AI's reach continues to grow, evaluating its capability to understand physical laws, human guidelines, and hard constraints grows increasingly important. Engineering product design lies at the intersection of these difficult tasks, providing new challenges for AI capabilities. BikeBench evaluates AI models' capabilities to generate bicycle designs that not only resemble the dataset, but meet specific performance objectives and constraints. To do so, BikeBench quantifies a variety of human-centered and multiphysics performance characteristics, such as aerodynamics, ergonomics, structural mechanics, human-rated usability, and similarity to subjective text or image prompts. Supporting the benchmark are several datasets of simulation results, a dataset of 10,000 human-rated bicycle assessments, and a synthetically generated dataset of 1.6M designs, each with a parametric, CAD/XML, SVG, and PNG representation. BikeBench is uniquely configured to evaluate tabular generative models, large language models (LLMs), design optimization, and hybrid algorithms side-by-side. Our experiments indicate that LLMs and tabular generative models fall short of hybrid GenAI+optimization algorithms in design quality, constraint satisfaction, and similarity scores, suggesting significant room for improvement. We hope that BikeBench, a first-of-its-kind benchmark, will help catalyze progress in generative AI for constrained multi-objective engineering design problems. We provide code, data, an interactive leaderboard, and other resources at https://github.com/Lyleregenwetter/BikeBench.

Crowdsourcing in China is a thriving industry. Among its most interesting structures, we find crowdfarms, in which crowdworkers self-organize as small organizations to tackle macrotasks. Little, however, is known as to which practices these crowdfarms use to tackle the macrotasks, and this goes hand in hand with the current practice of the HCI research community to treat all forms of complex crowdsourcing work as practically the same. However, macrotasks differ substantially regarding structure and decomposability. Treating them under one umbrella term - macrotasking - can lead to an imprecise understanding of the workforce involved. We address this gap by examining the work practices of 31 Chinese crowdfarms on the four main macrotask types, namely: modular, interlaced, wicked, and container macrotasks. Our results confirm essential differences in how these nascent crowd organizations address different macrotasks and shed light on what platforms can do to improve the uptake of such work.

Self-assembling team formation systems, where online users can select their teammates, are gaining research and industry interest. Still, the benefits of diversity remain frequently untapped for these teams, as people tend to choose others similar to them. In this study, we examine whether making users aware of the team’s diversity can impact their selections. In a study involving 120 crowd participants, working on the scenario of a crowdsourced innovation project, we tested the effects of two choice architecture and nudging techniques. The first technique displayed explicit personalized diversity information in the form of the current team diversity score and diversity recommendations. The second technique used diversity priming, in the form of counter-stereotypes and all-inclusive multiculturalism. Our results indicate that, while priming deterred participants from picking teammates of different regions, displaying diversity information was the only factor to positively enhance diverse choices. These results were not moderated by the users’ ’‘need to belong” levels, an intrinsic motivation justifying one’s need to form social ties. Other factors which we also find to predict selection behavior were the participants’ region of origin, participants’ gender, teammates’ functional backgrounds, and teammates’ order of appearance. In light of these findings, we suggest that nudging techniques need to be cautiously applied to online team formation as the different techniques differ in their ability to evoke diversity among intrinsically diverse crowds, and that personalised displaying of diversity information seems most promising.

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers.

Recent research highlights the potential of crowdsourcing in China. Yet very few studies explore the workplace context and experiences of Chinese crowdworkers. Those that do, focus mainly on the work experiences of solo crowdworkers but do not deal with issues pertaining to the substantial amount of people working in 'crowdfarms'. This article addresses this gap as one of its primary concerns. Drawing on a study that involves 48 participants, our research explores, compares and contrasts the work experiences of solo crowdworkers to those of crowdfarm workers. Our findings illustrate that the work experiences and context of the solo workers and crowdfarm workers are substantially different, with regards to their motivations, the ways they engage with crowdsourcing, the tasks they work on, and the crowdsourcing platforms they utilize. Overall, our study contributes to furthering the understandings on the work experiences of crowdworkers in China.

Despite the growing interest in crowdsourcing, this new labor model has recently received severe criticism. The most important point of this criticism is that crowdworkers are often underpaid and overworked. This severely affects job satisfaction and productivity. Although there is a growing body of evidence exploring the work experiences of crowdworkers in various countries, there have been a very limited number of studies to the best of our knowledge exploring the work experiences of Chinese crowdworkers. In this paper we aim to address this gap. Based on a framework of well-established approaches, namely the Job Demands-Resources model, the Work Design Questionnaire, the Oldenburg Burnout Inventory, the Utrecht Work Engagement Scale, and the Organizational Commitment Questionnaire, we systematically study the work experiences of 289 crowdworkers who work for ZBJ.com - the most popular Chinese crowdsourcing platform. Our study examines these crowdworker experiences along four dimensions: (1) crowdsourcing job demands, (2) job resources available to the workers, (3) crowdwork experiences, and (4) platform commitment. Our results indicate significant differences across the four dimensions based on crowdworkers' gender, education, income, job nature, and health condition. Further, they illustrate that different crowdworkers have different needs and threshold of demands and resources and that this plays a significant role in terms of moderating the crowdwork experience and platform commitment. Overall, our study sheds light to the work experiences of the Chinese crowdworkers and at the same time contributes to furthering understandings related to the work experiences of crowdworkers.

Expressions of territoriality have been positioned as one of the main reasons users alter their behaviors and perceptions of spatiality and sociality while engaging with location-based social networks (LBSN). Despite the potential for this interplay to further our understanding of LBSN usage in the context of identity, very little work has actually been done toward this. Addressing this gap in the literature is one of the chief aims of the article. Drawing on an original 6-week study with 42 participants utilizing a bespoke LBSN entitled “GeoMoments,” our research explores the following: (1) the way that territoriality is linked to self-identity; and (2) how this interplay affects the interactions between users as well as the environments they inhabit. Our findings suggest that participants affirmed their self-identity by selectively posting and claiming ownership of their neighborhood through the LBSN. Here, the locative decisions are made related to risk, hierarchies, and the users’ relationship to the area. This practice then led participants to discover and interact with the digital information overlaying their physical environments in a playful manner. These interactions demonstrate the perceived power structures that are facilitated by identity claims over a virtual area. In the main, our results reaffirm that territoriality is a central concept in understanding LBSN use, while also drawing attention to the temporality involved in user-to-user and user-to-place interactions pertaining to physical place mediated by LBSN.

Following successful crowd ideation contests, organizations in search of the "next big thing" are left with hundreds of ideas. Expert-based idea filtering is lengthy and costly; therefore, crowd-based strategies are often employed. Unfortunately, these strategies typically (1) do not separate the mediocre from the excellent, and (2) direct all the attention to certain idea concepts, while others starve. We introduce DBLemons - a crowd-based idea filtering strategy that addresses these issues by (1) asking voters to identify the worst rather than the best ideas using a "bag of lemons'' voting approach, and (2) by exposing voters to a wider idea spectrum, thanks to a dynamic diversity-based ranking system balancing idea quality and coverage. We compare DBLemons against two state-of-the-art idea filtering strategies in a real-world setting. Results show that DBLemons is more accurate, less time-consuming, and reduces the idea space in half while still retaining 94% of the top ideas.

Forming work teams involves matching people with complementary skills and personalities, but requires obtaining such data a priori. We introduce team dating, where people interact on brief tasks before working with a dedicated partner for longer, more complex tasks. We studied team dating through two online experiments. In Experiment 1, workers from a crowd platform independently wrote an ad slogan, discussed it with three consecutive people and evaluated their team date interactions. They then selected preferred teammates from a list showing average ratings for people they had dated and not dated. Results show that participants evaluated their dates based on evidence beyond externally judged slogan quality, and relied heavily on their dyad-specific judgments in selecting teammates. In Experiment 2, we replicated the individual and team dating tasks, and formed teams, either i) by honoring pairwise team dating preferences, ii) randomly from their pool of dates, or iii) randomly from those not dated. Results show that teams formed from preferred dates performed better on a final creative task compared to random dates or non-dates. Team dating provides a dynamic technique for forming ad hoc teams accounting for interpersonal dynamics. The initial interactions provided information that helped people select and work with an appropriate teammate.

When personalities clash, teams operate less effectively. Personality differences affect face-to-face collaboration and may lower trust in virtual teams. For relatively short-lived assignments, like those of online crowdsourcing, personality matching could provide a simple, scalable strategy for effective team formation. However, it is not clear how (or if) personality differences affect teamwork in this novel context where the workforce is more transient and diverse. This study examines how personality compatibility in crowd teams affects performance and individual perceptions. Using the DISC personality test, we composed 14 five-person teams (N=70) with either a harmonious coverage of personalities (balanced) or a surplus of leader-type personalities (imbalanced). Results show that balancing for personality leads to significantly better performance on a collaborative task. Balanced teams exhibited less conflict and their members reported higher levels of satisfaction and acceptance. This work demonstrates a simple personality matching strategy for forming more effective teams in crowdsourcing contexts.

In this paper the authors propose a crowd coordination mechanism to increase the quality of articles produced in wiki systems. Wikis constitute promising social digital ecosystems for collaborative knowledge creation on the Web. However, as a result of the purely self-coordinated manner that they function, they cannot ensure the quality of the produced articles - an issue that affects their reliability and acceptance. The authors show that wiki article quality optimization can be formulated as a resource allocation problem. Contributors are selected from the wiki community crowd according to their skills, and matched to the articles they can improve the most. A model of the English Wikipedia is given, parameterized and validated from recent field studies results. Experimental results were obtained with simulation systems implementing this model and on a series of scenarios, which include an analysis of the impact of using semantic relations between wiki domains. The obtained results indicate that the proposed mechanism can lead to the production of wiki articles of higher quality, compared to the respective results achieved by the fully self-coordinated wiki.

In this paper, we present a framework for engineering collective intelligence systems that will be used by web communities. The proposed framework enables the development of communitydriven, self-regulating CI systems, which adapt their functionality to the activity and goals of the web community. The above engineering methodology is applied on the design of a popular web system, namely Wikipedia, to illustrate the way that the functionality of the latter could be improved, in terms of better and more prompt article quality production. The preliminary evaluation results of this application, obtained through simulation modeling are promising. Copyright 2010 ACM.

Collective intelligence (CI) is an emerging research field which aims at combining human and machine intelligence, to improve community processes usually performed by large groups. CI systems may be collaborative, like Wikipedia, or competitive, like a number of recently established problem-solving companies that attempt to find solutions to difficult R&D or marketing problems drawing on the competition among web users. The benefits that CI systems earn user communities, combined with the fact that they share a number of basic common characteristics, open up the prospect for the design of a general methodology that will allow the efficient development and evaluation of CI. In the present work, an attempt is made to establish the analytical foundations and main challenges for the design and construction of a generic collective intelligence system. First, collective intelligence systems are categorized into active and passive and specific examples of each category are provided. Then, the basic modeling framework of CI systems is described. This includes concepts such as the set of possible user actions, the CI system state and the individual and community objectives. Additional functions, which estimate the expected user actions, the future state of the system, as well as the level of objective fulfillment, are also established. In addition, certain key issues that need to be considered prior to system launch are also described. The proposed framework is expected to promote efficient CI design, so that the benefit gained by the community and the individuals through the use of CI systems, will be maximized. Copyright 2009 ACM.

Abstract The increasing popularity of e‐learning has created a need for accurate student achievement prediction mechanisms, allowing instructors to improve the efficiency of their courses by addressing specific needs of their students at an early stage. In this paper, a student achievement prediction method applied to a 10‐week introductory level e‐learning course is presented. The proposed method uses multiple feed‐forward neural networks to dynamically predict students' final achievement and to cluster them in two virtual groups, according to their performance. Multiple‐choice test grades were used as the input data set of the networks. This form of test was preferred for its objectivity. Results showed that accurate prediction is possible at an early stage, more specifically at the third week of the 10‐week course. In addition, when students were clustered, low misplacement rates demonstrated the adequacy of the approach. The results of the proposed method were compared against those of linear regression and the neural‐network approach was found to be more effective in all prediction stages. The proposed methodology is expected to support instructors in providing better educational services as well as customized assistance according to students' predicted level of performance.