How can society understand and hold accountable complex human and algorithmic decision-making systems whose systematic errors are opaque to the public? These systems routinely make decisions on individual rights and well-being, and on protecting society and the democratic process. Practical and statistical constraints on external audits--such as dimensional complexity--can lead researchers and regulators to miss important sources of error in these complex decision-making systems. In this paper, we design and implement a software-supported approach to audit studies that auto-generates audit materials and coordinates volunteer activity. We implemented this software in the case of political advertising policies enacted by Facebook and Google during the 2018 U.S. election. Guided by this software, a team of volunteers posted 477 auto-generated ads and analyzed the companies' actions, finding systematic errors in how companies enforced policies. We find that software can overcome some common constraints of audit studies, within limitations related to sample size and volunteer capacity.

As online platforms monitor and intervene in the daily lives of billions of people, platforms are being used to govern enduring social problems. Field experiments could inform wise uses of this power if tensions between democratic values and experimentation could be resolved. In this paper, we introduce CivilServant, a novel experimentation infrastructure that online communities and their moderators use to evaluate policies and replicate each others' findings. We situate CivilServant in the political history of policy experiments and present design considerations for community participation, ethics, and replication. Based on two case studies of community-led experiments and public debriefings on the reddit platform, we share findings on community deliberation about experiment results. We also report on uses of evidence, finding that experiments informed moderator practices, community policies, and replications by communities and platforms. We discuss the implications of these findings for evaluating platform governance in an open, democratic, experimenting society.

Networked gatekeepers on social media increasingly influence which people and groups receive media attention. Many unknowingly direct much greater attention to men than to women. Can technologies support these gatekeepers to follow their own values of equality? Theories of value consistency suggest that confronting people with inconsistencies between values and behavior can prompt behavior change. In this paper, we introduce FollowBias, a novel system that offers feedback on the percentage of women that users follow onTwitter. We conduct field deployments of FollowBias with 61 and 78 participants, exploring differences between their values and behavior, their explanations of those differences, and their changes in behavior. In the first, FollowBias users had a 45 percentage point greater chance of increasing the percentage of women followed over one week. In the second, we fail to find an effect. We also offer findings on political and ethical trade-offs in designing systems for behavior change toward equality.

This paper describes how people who lead communities on online platforms join together in mass collective action to influence platform operators. I investigate this by analyzing a protest against the social news platform reddit by moderators of 2,278 subreddit communities in July 2015. These moderators collectively disabled their subreddits, preventing millions of readers from accessing major parts of reddit and convincing the company to negotiate over their demands. This paper offers a descriptive analysis of the protest, combining qualitative content analysis, interviews, and quantitative analysis with the population of 52,735 active subreddits. Through participatory hypotheses testing with moderators, this study reveals social factors including the grievances of moderators, relations with platform operators, relations among moderators, subreddit resources, subreddit isolation, and moderators' relations with their subreddits that can lead to participation in mass collective action against a platform.

This paper contributes to a growing body of work that attempts to measure informal learning online by revisiting two of the most surprising findings from a 2012 study on skill progression in Scratch by Scaffidi and Chambers: users tend to share decreasingly code-heavy projects over time; and users' projects trend toward using a less diverse range of code concepts. We revisit Scaffidi and Chambers's work in three ways: with a replication of their study using the full population of projects from which they sampled, a simulation study that replicates both their analytic and sampling methodology, and an alternative analysis that addresses several important threats. Our results suggest that the population estimates are opposite in sign to those presented in the original work.

In addition to posting news and status updates, many Twitter users post questions that seek various types of subjective and objective information. These questions are often labeled with 'Q&A' hashtags, such as #lazyweb or #twoogle. We surveyed Twitter users and found they employ these Q&A hashtags both as a topical signifier (this tweet needs an answer!) and to reach out to those beyond their immediate followers (a community of helpful tweeters who monitor the hashtag). However, our log analysis of thousands of hashtagged Q&A exchanges reveals that nearly all replies to hashtagged questions come from a user's immediate follower network, contradicting users' beliefs that they are tapping into a larger community by tagging their question tweets. This finding has implications for designing next-generation social search systems that reach and engage a wide audience of answerers.