Privacy concerns significantly impact AI adoption, yet little is known about how information environments shape user responses to data leak threats. We conducted a 2 × 3 between-subjects experiment (N = 610) examining how risk versus ambiguity about privacy leaks affects the adoption of AI personalization. Participants chose between standard and AI-personalized product baskets, with personalization requiring data sharing that could leak to pricing algorithms. Under risk (30% leak probability), we found no difference in AI adoption between privacy-threatening and neutral conditions (ca. 50% adoption). Under ambiguity (10-50% range), privacy threats significantly reduced adoption compared to neutral conditions. This effect holds for sensitive demographic data as well as anonymized preference data. Users systematically over-bid for privacy disclosure labels, suggesting strong demand for transparency institutions. Notably, privacy leak threats did not affect subsequent bargaining behavior with algorithms. Our findings indicate that ambiguity over data leaks, rather than only privacy preferences per se, drives avoidance behavior among users towards personalized AI.

Pilot studies are an essential cornerstone of the design of crowdsourcing campaigns, yet they are often only mentioned in passing in the scholarly literature. A lack of details surrounding pilot studies in crowdsourcing research hinders the replication of studies and the reproduction of findings, stalling potential scientific advances. We conducted a systematic literature review on the current state of pilot study reporting at the intersection of crowdsourcing and HCI research. Our review of ten years of literature included 171 articles published in the proceedings of the Conference on Human Computation and Crowdsourcing (AAAI HCOMP) and the ACM Digital Library. We found that pilot studies in crowdsourcing research (i.e., crowd pilot studies) are often under-reported in the literature. Important details, such as the number of workers and rewards to workers, are often not reported. On the basis of our findings, we reflect on the current state of practice and formulate a set of best practice guidelines for reporting crowd pilot studies in crowdsourcing research. We also provide implications for the design of crowdsourcing platforms and make practical suggestions for supporting crowd pilot study reporting.

A wealth of evidence across several domains indicates that goal setting improves performance and learning by enabling individuals to commit their thoughts and actions to goal achievement. Recently, researchers have begun studying the effects of goal setting in paid crowdsourcing to improve the quality and quantity of contributions, increase learning gains, and hold participants accountable for contributing more effectively. However, there is a lack of research addressing crowd workers' goal-setting practices, how they are currently pursuing them, and the challenges that they face. This information is essential for researchers and developers to create tools that assist crowd workers in pursuing their goals more effectively, thereby improving the quality of their contributions. This paper addresses these gaps by conducting mixed-method research in which we surveyed 205 workers from two crowdsourcing platforms -- Amazon Mechanical Turk (MTurk) and Prolific -- about their goal-setting practices. Through a 14-item survey, we asked workers regarding the types of goals they create, their goal achievement strategies, potential barriers that impede goal attainment, and their use of software tools for effective goal management. We discovered that (a) workers actively create intrinsic and extrinsic goals; (b) use a combination of tools for goal management; (c) medical issues and a busy lifestyle are some obstacles to their goal achievement; and (d) we gathered novel features for future goal management tools. Our findings shed light on the broader implications of developing goal management tools to improve workers' well-being.

Crowd-powered conversational systems (CPCS) are gaining considerable attention for their potential utility in a variety of application domains, for which automated conversational interfaces are still too limited. CPCS currently suffer from long response delays, which hampers their potential as conversational partners. The majority of prior work in this area has focused on demonstrating the feasibility of the approach and improving performance, while evaluation studies have primarily focused on response latency and ways to reduce it. Relatively little is currently known about how response delays in a CPCS can affect user experience. While the importance of reducing response latency is widely recognized in the broader field of human-computer interaction, little attention has been paid to how response quality, response delay, conversational context, and the complexity of the task affect how users experience the conversation, and how they perceive waiting for responses in particular. We conducted a between-subjects experiment (N = 478), to examine the influence of these four factors on the overall waiting experience of users. Results show that users 1) evaluated the waiting experience more negatively when the response delay was longer than 8 seconds, 2) underestimated the elapsed time but experienced more frustration in tasks with high complexity, 3) underestimated the elapsed time and experienced less frustration with high quality bot's utterances, 4) judged response delays to be slightly longer, and experienced more frustration in an emotion-centric CPCS compared to a task-centric CPCS. Our insights can inform the design of future CPCSs with regards to defining performance requirements and anticipating their potential impact on the user experience they can facilitate.

Crowd-Powered Conversational Systems (CPCS) are gaining traction due to their potential utility in a range of application fields where automated conversational interfaces are still inadequate. Currently, long response times negatively impact CPCSs, limiting their potential application as conversational partners. Related research has focused on developing algorithms for swiftly hiring workers and synchronous crowd coordination techniques to ensure high-quality work. Evaluation studies typically concern system reaction times and performance measurements, but have so far not examined the effects of extended wait times on users. The goal of this study, based on time perception models, is to explore how effective different time fillers are at reducing the negative impacts of waiting in CPCSs. To this end, we conducted a rigorous simulation-based between-subjects (N = 930) study on the Prolific crowdsourcing platform to assess the influence of different filler types across three levels of delay (8, 16 & 32s) for Information Retrieval (IR) and stress management tasks. Our results show that asking users to perform secondary tasks (e.g., microtasks or breathing exercises) while waiting for longer periods of time helped divert their attention away from timekeeping, increased their engagement, and resulted in shorter perceived waiting times. For shorter delays, conversational fillers generated more intense immersion and contributed to shorten the perception of time.

On-demand emotional support is an expensive and elusive societal need that is exacerbated in difficult times — as witnessed during the COVID-19 pandemic. Prior work in affective crowdsourcing has examined ways to overcome technical challenges for providing on-demand emotional support to end users. This can be achieved by training crowd workers to provide thoughtful and engaging on-demand emotional support. Inspired by recent advances in conversational user interface research, we investigate the efficacy of a conversational user interface for training workers to deliver psychological support to users in need. To this end, we conducted a between-subjects experimental study on Prolific, wherein a group of workers (N=200) received training on motivational interviewing via either a conversational interface or a conventional web interface. Our results indicate that training workers in a conversational interface yields both better worker performance and improves their user experience in on-demand stress management tasks.

The IoT industry supplies a plethora of Internet connected devices and services supporting smart home automation. However, end-users having little knowledge of the features and possibilities of such technologies, face difficulties in conjuring up useful application scenarios combining such devices and services, thus missing out on potential applications outside those provided by vendors. A remedy for such end-users can potentially be found in crowdsourcing IoT scenario creation. For such an enterprise to be viable it is essential to assess whether crowdsourcing can result in practical and original scenarios. This article reports two studies aiming to establish the practicality and originality of crowdsourced IoT scenarios for smart homes. In the first study, we recruited 102 crowd workers who created 306 scenarios in various categories. We then recruited a second cohort of 620 crowd workers to rate the scenarios’ creativity. In the second study, we evaluated the corpus of IoT scenarios by 20-experienced smart home users recruited through a screening survey. Our results show that the crowd evaluations of originality and creativity are strongly correlated with those of smart home users. Our major IoT-specific findings in relation to creativity are: a) The number of IoT devices and the number of combination of devices impact how creative the scenarios are perceived; b) Workers with self-reported intermediate programming knowledge wrote more creative scenarios when compared to workers having expert knowledge; c) Computational metrics such as text metrics can provide the basis for automated assessment of the scenarios’ creativity. Finally, an inductive thematic analysis of the scenarios revealed interesting themes (e.g., types of rules, automation styles and novel operators) which can serve as a guide for designing more expressive and intuitive end-user development solutions, in the context of IoT.