As companies increasingly rely on experiments to make product decisions, precisely measuring changes in key metrics is important. Various methods to increase sensitivity in experiments have been proposed, including methods that use pre-experiment data, machine learning, and more advanced experimental designs. However, prior work has not explored modeling heterogeneity in the variance of individual experimental users. We propose a more sensitive treatment effect estimator that relies on estimating the individual variances of experimental users using pre-experiment data. We show that that weighted estimators using individual-level variance estimates can reduce the variance of treatment effect estimates, and prove that the coefficient of variation of the sample population variance is a sufficient statistic for determining the scale of possible variance reduction. We provide empirical results from case studies at Facebook demonstrating the effectiveness of this approach, where the average experiment achieved a 17% reduction in variance with minimal impact on bias.

Public discussions on social media platforms are an intrinsic part of online information consumption. Characterizing the diverse range of discussions which can arise is crucial for these platforms, as they may seek to organize and curate them. This paper introduces a framework to characterize public discussions, relying on a representation that captures a broad set of social patterns which emerge from the interactions between interlocutors, comments, and audience reactions. We apply our framework to study public discussions on Facebook at two complementary scales. First, at the level of individual discussions, we use it to predict a discussion's future trajectory, anticipating future antisocial actions (such as participants blocking each other) and forecasting the discussion's growth. Second, we systematically analyze the variation of discussions across thousands of Facebook sub-communities, revealing subtle differences (and unexpected similarities) in how people interact when discussing online content. We further show that this variation is driven more by participant tendencies than by the content triggering these discussions.

Studies of online social influence have demonstrated that friends have important effects on many types of behavior in a wide variety of settings. However, we know much less about how influence works among relative strangers in digital public squares, despite important conversations happening in such spaces. We present the results of a study on large public Facebook Pages where we randomly used two different methods---most recent and social feedback---to order comments on posts. We find that the social feedback condition results in higher quality viewed comments and response comments. After measuring the average quality of comments written by users before the study, we find that social feedback has a positive effect on response quality for both low and high quality commenters. We draw on a theoretical framework of social norms to explain this empirical result. In order to examine the influence mechanism further, we measure the similarity between comments viewed and written during the study, finding that similarity increases for the highest quality contributors under the social feedback condition. This suggests that, in addition to norms, some individuals may respond with increased relevance to high-quality comments.

Most models of social contagion take peer exposure to be a corollary of adoption, yet in many settings, the visibility of one's adoption behavior happens through a separate decision process. In online systems, product designers can define how peer exposure mechanisms work: adoption behaviors can be shared in a passive, automatic fashion, or occur through explicit, active sharing. The consequences of these mechanisms are of substantial practical and theoretical interest: passive sharing may increase total peer exposure but active sharing may expose higher quality products to peers who are more likely to adopt.