Most generative AI tools prioritize individual productivity and personalization, with limited support for collaboration. Designed for traditional workplaces, these tools do not fit freelancers’ short-term teams or lack of shared institutional support, which can worsen their isolation and overlook freelancing platform dynamics. This mismatch means that, instead of empowering freelancers, current generative AI tools could reinforce existing precarity and make freelancer collaboration harder. To investigate how to design generative AI tools to support freelancer collaboration, we conducted co-design sessions with 27 freelancers. A key concern that emerged was the risk of AI systems compromising their creative agency and work identities when collaborating, especially when AI tools could reproduce content without attribution, threatening the authenticity and distinctiveness of their collaborative work. Freelancers proposed "auxiliary AI" systems, human-guided tools that support their creative agencies and identities, allowing for flexible freelancer-led collaborations that promote "productive friction". Drawing on Marcuse’s concept of technological rationality, we argue that freelancers are resisting one-dimensional, efficiency-driven AI, and instead envisioning technologies that preserve their collective creative agencies. We conclude with design recommendations for collaborative generative AI tools for freelancers.

AI is not only a neutral tool in team settings; it influence the social and cognitive fabric of collaboration. Across two randomized experiments, we demonstrate that AI exposure produces causal spillover into human–human interaction—affecting shared language, collective attention, shared mental models, and social cohesion. These spillover effects occur robustly across settings, modalities, tasks, and AI qualities, suggesting that mere exposure to AI drives the influence. AI functions as an implicit “social forcefield,” influencing not only how people speak, but also how they think, what they attend to, and how they relate to each other. We argue for shifting the design paradigm from optimizing “AI as a tool” to understanding AI as a socially influential actor whose effects extend beyond the human–AI interface.

The rapid adoption of generative AI in software development has impacted the industry, yet its effects on developers with visual impairments remain largely unexplored. To address this gap, we used an Activity Theory framework to examine how developers with visual impairments interact with AI coding assistants. For this purpose, we conducted a study where developers who are visually impaired completed a series of programming tasks using a generative AI coding assistant. We uncovered that, while participants found the AI assistant beneficial and reported significant advantages, they also highlighted accessibility challenges. Specifically, the AI coding assistant often exacerbated existing accessibility barriers and introduced new challenges. For example, it overwhelmed users with an excessive number of suggestions, leading developers who are visually impaired to express a desire for ``AI timeouts.'' Additionally, the generative AI coding assistant made it more difficult for developers to switch contexts between the AI-generated content and their own code. Despite these challenges, participants were optimistic about the potential of AI coding assistants to transform the coding experience for developers with visual impairments. Our findings emphasize the need to apply activity-centered design principles to generative AI assistants, ensuring they better align with user behaviors and address specific accessibility needs. This approach can enable the assistants to provide more intuitive, inclusive, and effective experiences, while also contributing to the broader goal of enhancing accessibility in software development.

As independently-contracted employees, gig workers disproportionately suffer the consequences of workplace surveillance, which include increased pressures to work, breaches of privacy, and decreased digital autonomy. Despite the negative impacts of workplace surveillance, gig workers lack the tools, strategies, and workplace social support to protect themselves against these harms. Meanwhile, some critical theorists have proposed sousveillance as a potential means of countering such abuses of power, whereby those under surveillance monitor those in positions of authority (e.g., gig workers collect data about requesters/platforms). To understand the benefits of sousveillance systems in the gig economy, we conducted semi-structured interviews and led co-design activities with gig workers. We use “care ethics” as a guiding concept to understand our interview and co-design data, while also focusing on empathic sousveillance technology design recommendations. Through our study we identify gig workers’ attitudes towards and past experiences with sousveillance. We also uncover the type of sousveillance technologies imagined by workers, provide design recommendations, and finish by discussing how to create empowering, empathic spaces on gig platforms.

Collective action by gig knowledge workers is a potent method for enhancing labor conditions on platforms like Upwork, Amazon Mechanical Turk, and Toloka. However, this type of collective action is still rare today. Existing systems for supporting collective action are inadequate for workers to identify and understand their different workplace problems, plan effective solutions, and put the solutions into action. This talk will discuss how with my research lab we are creating worker-centric AI enhanced technologies that enable collective action among gig knowledge workers. Building solid AI enhanced technologies to enable gig worker collective action will pave the way for a fair and ethical gig economy-one with fair wages, humane working conditions, and increased job security. I will discuss how my proposed approach involves first integrating ''sousveillance,'' a concept by Foucault, into the technologies. Sousveillance involves individuals or groups using surveillance tools to monitor and record those in positions of power. In this case, the technologies enable gig workers to monitor their workplace and their algorithmic bosses, giving them access to their own workplace data for the first time. This facilitates the first stage of collective action: problem identification. I will then discuss how we combine this data with Large-Language-Models (LLMs) and social theories to create intelligent assistants that guide workers to complete collective action via sensemaking and solution implementation.

Crowdsourcing markets are expanding worldwide, but often feature standardized interfaces that ignore the cultural diversity of their workers, negatively impacting their well-being and productivity. To transform these workplace dynamics, this paper proposes creating culturally-aware workplace tools, specifically designed to adapt to the cultural dimensions of monochronic and polychronic work styles. We illustrate this approach with "CultureFit," a tool that we engineered based on extensive research in Chronemics and culture theories. To study and evaluate our tool in the real world, we conducted a field experiment with 55 workers from 24 different countries. Our field experiment revealed that CultureFit significantly improved the earnings of workers from cultural backgrounds often overlooked in design. Our study is among the pioneering efforts to examine culturally aware digital labor interventions. It also provides access to a dataset with over two million data points on culture and digital work, which can be leveraged for future research in this emerging field. The paper concludes by discussing the importance and future possibilities of incorporating cultural insights into the design of tools for digital labor.

Crowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles. We welcome the studies on developing and enhancing of crowdworker-centric tools, that offer task matching, requester assessment, instruction validation, among other topics. We are also interested in exploring methods that leverage the integration of crowdworkers to improve the recognition and performance of the machine learning models. Thus, we invite studies that focus on shipping active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine.

The limited information (data voids) on political topics relevant to underrepresented communities has facilitated the spread of disinformation. Independent journalists who combat disinformation in underrepresented communities have reported feeling overwhelmed because they lack the tools necessary to make sense of the information they monitor and address the data voids. In this paper, we present a system to identify and address political data voids within underrepresented communities. Armed with an interview study, indicating that the independent news media has the potential to address them, we designed an intelligent collaborative system, called Datavoidant. Datavoidant uses state-of-the-art machine learning models and introduces a novel design space to provide independent journalists with a collective understanding of data voids to facilitate generating content to cover the voids. We performed a user interface evaluation with independent news media journalists (N=22). These journalists reported that Datavoidant's features allowed them to more rapidly while easily having a sense of what was taking place in the information ecosystem to address the data voids. They also reported feeling more confident about the content they created and the unique perspectives they had proposed to cover the voids. We conclude by discussing how Datavoidant enables a new design space wherein individuals can collaborate to make sense of their information ecosystem and actively devise strategies to prevent disinformation.

Crowdworkers depend on Amazon Mechanical Turk (AMT) as an important source of income and it is left to workers to determine which tasks on AMT are fair and worth completing. While there are existing tools that assist workers in making these decisions, workers still spend significant amounts of time finding fair labor. Difficulties in this process may be a contributing factor in the imbalance between the median hourly earnings ($2.00/hour) and what the average requester pays ($11.00/hour). In this paper, we study how novices and experts select what tasks are worth doing. We argue that differences between the two populations likely lead to the wage imbalances. For this purpose, we first look at workers' comments in TurkOpticon (a tool where workers share their experience with requesters on AMT). We use this study to start to unravel what fair labor means for workers. In particular, we identify the characteristics of labor that workers consider is of "good quality'' and labor that is of "poor quality'' (e.g., work that pays too little.) Armed with this knowledge, we then conduct an experiment to study how experts and novices rate tasks that are of both good and poor quality. Through our research we uncover that experts and novices both treat good quality labor in the same way. However, there are significant differences in how experts and novices rate poor quality labor, and whether they believe the poor quality labor is worth doing. This points to several future directions, including machine learning models that support workers in detecting poor quality labor, and paths for educating novice workers on how to make better labor decisions on AMT.

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.

Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural Amazon Mechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of 421 crowd workers from differing geographic regions in the U.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools.

Researchers in various fields have been discussing the ethics of field research, particularly their responsibility to provide concrete benefits to participants. For example, when designing technology for their participants the discussion has centered around how and whether the technology benefits participants. We argue, that design projects can be reoriented towards benefiting participants from the outset, by slightly changing the process, activities, and initial object of design. Focusing instead on how to immediately impact the goals of the participants, designers can then progressively introduce technology to scale their impact. In this paper we present an ongoing instantiation of this process in rural West Virginia, where we have been teaching computer classes at a local library. We have found that our participants and partners are more appreciative of our efforts, and that the data we are gathering is just as, if not more, rich than previous design methodologies that we have used.

Crowd markets have traditionally limited workers by not providing transparency information concerning which tasks pay fairly or which requesters are unreliable. Researchers believe that a key reason why crowd workers earn low wages is due to this lack of transparency. As a result, tools have been developed to provide more transparency within crowd markets to help workers. However, while most workers use these tools, they still earn less than minimum wage. We argue that the missing element is guidance on how to use transparency information. In this paper, we explore how novice workers can improve their earnings by following the transparency criteria of Super Turkers, i.e., crowd workers who earn higher salaries on Amazon Mechanical Turk (MTurk). We believe that Super Turkers have developed effective processes for using transparency information. Therefore, by having novices follow a Super Turker criteria (one that is simple and popular among Super Turkers), we can help novices increase their wages. For this purpose, we: (i) conducted a survey and data analysis to computationally identify a simple yet common criteria that Super Turkers use for handling transparency tools; (ii) deployed a two-week field experiment with novices who followed this Super Turker criteria to find better work on MTurk. Novices in our study viewed over 25,000 tasks by 1,394 requesters. We found that novices who utilized this Super Turkers’ criteria earned better wages than other novices. Our results highlight that tool development to support crowd workers should be paired with educational opportunities that teach workers how to effectively use the tools and their related metrics (e.g., transparency values). We finish with design recommendations for empowering crowd workers to earn higher salaries.

Our study presents a new tool, Reputation Agent, to promote fairer reviews from requesters (employers or customers) on gig markets. Unfair reviews, created when requesters consider factors outside of a worker’s control, are known to plague gig workers and can result in lost job opportunities and even termination from the marketplace. Our tool leverages machine learning to implement an intelligent interface that: (1) uses deep learning to automatically detect when an individual has included unfair factors into her review (factors outside the worker’s control per the policies of the market); and (2) prompts the individual to reconsider her review if she has incorporated unfair factors. To study the effectiveness of Reputation Agent, we conducted a controlled experiment over different gig markets. Our experiment illustrates that across markets, Reputation Agent, in contrast with traditional approaches, motivates requesters to review gig workers’ performance more fairly. We discuss how tools that bring more transparency to employers about the policies of a gig market can help build empathy thus resulting in reasoned discussions around potential injustices towards workers generated by these interfaces. Our vision is that with tools that promote truth and transparency we can bring fairer treatment to gig workers.

While past work has admirably supported crowd workers in improving their work performance, we argue that there is also value in designing for enjoyment untied from work outcomes--- what we call "tangential play.'' To this end, we present Turker Tales, a Google Chrome extension that uses tangential play to encourage crowd workers to write, share, and view short tales as a side activity to their main job on Amazon Mechanical Turk (MTurk). Turker Tales introduces a layer of playful narrativization atop typical crowd work tasks in order to alter workers' experiences of those tasks without aiming to improve work efficiency or quality. Using speed-dating (N=12) and a pilot test (N=150) to inform our design, we deployed Turker Tales over one week with 171 participants, receiving 1,096 tales and 1,527 ratings of those tales. We found that our system of tangential play brought to light underlying conflicts (such as unfair working conditions), and provided a space for participants to reveal aspects of themselves and their shared experiences. Through Turker Tales, we critically reflect on the roles of researchers, designers, and requesters in crowd work, and the ethics of incorporating play into crowd work, and consider the implications of the paradigm we introduce both as a method of research through design and as a direction for design to support crowd workers.

Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ~ 150 derived features, and was able to predict the hourly wages of 69.6% of all the tested microtasks within a 75% error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences.)

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work.

Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.

Bots, or programs designed to engage in social spaces and perform automated tasks, are typically understood as automated tools or as social "chatbots." In this paper, we consider their place alongside users in the emerging social ecosystem of audience participation platforms, through the application of Structural Role Theory. We perform a large-scale analysis of activity levels of user-designed bots on Twitch, finding that they communicate at a much greater rate than any other type of user. We build on a classification scheme derived from prior literature on bot functionalities to identify the roles bots play on Twitch, how these roles vary across different types of Twitch communities, and how users engage with them and vice versa. We conclude with a discussion of what roles are missing and where opportunities lie to re-conceptualize and re-design bots as social actors who help communities grow and evolve.

Audience Participation Games challenge traditional assumptions about gameplay by blurring the line between audience and player, allowing audience members to impact gameplay in a meaningful way. Their recent rise in popularity has created new opportunities for game research and development. To better understand this design space, we developed several versions of two prototype games as design probes. We livestreamed them to an online audience in order to develop a framework for audience motivations and participation styles, to explore ways in which mechanics can affect audience members' sense of agency, and to identify promising design spaces. Our results show the breadth of opportunities and challenges that designers face in creating engaging Audience Participation Games.

Smart devices connected to the Internet are proliferating.To reduce costs of devices that havetraditionally been inexpensive(toasters, microwaves, printers, etc), manyof these devices have chosen to use a speech interface rather than a visual one. This transition has been hastened by the increasing capabilities of speech interfaces,exemplifiedbyproducts likeAmazon Echo and Apple'sSiri.A consequence of these products moving to voice control is that people who are deaf and hard of hearing (DHH) may be unable to use them. In this paper, we briefly introduce two technical approaches we are pursuingfor enabling DHH people to provide input to these devices: (i) human computationworkflows for understanding "deaf speech," and (ii) mobile interfaces that can be instructed to speak on the user's behalf.

Web accessibility and usability have been extensively studied for blind web users. The focus has generally been on making it technically possible for blind users to access content, or on helping to make the web more usable. This paper explores a challenge at the intersection of these two lenses, which is the effects of blind web users not knowing what they don't know. On the web, this often means that the user is having a problem completing a task, but does not know whether the problem is because the information is there and not accessible, whether the information is simply difficult to access, or whether the information is not present at all. We first discuss how this issue has manifested itself in other work in this space. We then present the results of a study with 30 sighted web users and 30 blind web users exploring the phenomenon, demonstrating that not knowing the source of a problem causes frustration and wastes time. We conclude with recommendations for future research to help understand and address this problem, as well as design implications for future technology that may assist non-visual web navigation.

Mainstream crowdwork platforms treat microtasks as indivisible units; however, in this article, we propose that there is value in re-examining this assumption. We argue that crowdwork platforms can improve their value proposition for all stakeholders by supporting subcontracting within microtasks. After describing the value proposition of subcontracting, we then define three models for microtask subcontracting: real-time assistance, task management, and task improvement, and reflect on potential use cases and implementation considerations associated with each. Finally, we describe the outcome of two tasks on Mechanical Turk meant to simulate aspects of subcontracting. We reflect on the implications of these findings for the design of future crowd work platforms that effectively harness the potential of subcontracting workflows.

To help activists call new volunteers to action, we present Botivist: a platform that uses Twitter bots to find potential volunteers and request contributions. By leveraging different Twitter accounts, Botivist employs different strategies to encourage participation. We explore how people respond to bots calling them to action using a test case about corruption in Latin America. Our results show that the majority of volunteers (80\%) who responded to Botivist's calls to action contributed relevant proposals to address the assigned social problem. Different strategies produced differences in the quantity and relevance of contributions. Some strategies that work well offline and face-to-face appeared to hinder people's participation when used by an online bot. We analyze user behavior in response to being approached by bots with an activist purpose. We also provide strong evidence for the value of this type of civic media, and derive design implications.

Armed groups of civilians known as "self-defense forces" have ousted the powerful Knights Templar drug cartel from several towns in Michoacán. This militia uprising has unfolded on social media, particularly in the "VXM" ("Valor por Michoacán," Spanish for "Courage for Michoacán") Facebook page, gathering more than 170,000 fans. Previous work on the Drug War has documented the use of social media for real-time reports of violent clashes. However, VXM goes one step further by taking on a pro-militia propagandist role, engaging in two-way communication with its audience. This paper presents a descriptive analysis of VXM and its audience. We examined nine months of posts, from VXM's inception until May 2014, totaling 6,000 posts by VXM administrators and more than 108,000 comments from its audience. We describe the main conversation themes, post frequency and relationships with offline events and public figures. We also characterize the behavior of VXM's most active audience members. Our work illustrates VXM's online mobilization strategies, and how its audience takes part in defining the narrative of this armed conflict. We conclude by discussing possible applications of our findings for the design of future communication technologies.

Disc Jokeys (DJs) generally mix music in a confined isolated space. This can make the DJ have depression sentiments and it can also difficult the DJ's understanding of his public. We present Enchantment Under The Sea: a new intelligent environment that allows the disc jokey to roam freely, interact directly with his audience, receive informative feedback about the public's social interactions, while also respecting the DJs privacy concerns. The music interface is controlled using Microsoft's wireless touch mouse with ubiquitous gestures that resemble dance moves. The music mixing interface is displayed on the walls of the event, where two different display modalities are enabled: open interface, in which the public can observe all of the DJ's decisions with the music mixing interface and also actively give music suggestions to the DJ. And a closed interface, where all the music controllers are mapped to sea animals, that only the DJ knows the mapping to, thus providing privacy to the DJ's work. The public's social interactions are measured with sonar sensors whose data is provided to the DJ through the musical interface. We report results of a controlled usability inspection.