OBJECTIVES: Digital health research involves collecting vast amounts of personal health data, making data management practices complex and challenging to convey during informed consent. MATERIALS AND METHODS: We conducted eight semi-structured focus groups to explore whether dataflow diagrams (DFD) can complement informed consent and improve participants' understanding of data management and associated risks (N = 34 participants). RESULTS: Our analysis found that DFDs could supplement text-based information about data management and sharing practices, such as by helping raise new questions that prompt conversation between prospective participants and members of a research team. Participants in the study emphasized the need for clear, simple, and accessible diagrams that are participant centered. Third-party access to data and sharing of sensitive health data were identified as high-risk areas requiring thorough explanation. Participants generally agreed that the design process should be led by the research team, but it should incorporate many diverse perspectives to ensure the diagram was meaningful to potential participants who are likely unfamiliar with data management. Nearly all participants rejected the idea that artificial intelligence could identify risks during the design process, but most were comfortable with it being used as a tool to format and simplify the diagram. In short, DFDs may complement standard text-based informed consent documents, but they are not a replacement. DISCUSSION: Prospective research participants value diverse ways of learning about study risks and benefits. Our study highlights the value of incorporating information visualizations, such as DFDs, into the informed consent procedures to participate in research. CONCLUSION: Future research should explore other ways of visualizing consent information in ways that help people to overcome digital and data literacy barriers to participating in research. However, creating a DFD requires significant time and effort from research teams. To alleviate these costs, research sponsors can support the creation of shared infrastructure, communities of practice, and incentivize researchers to develop better consent procedures.

Informed consent is a cornerstone of ethical human subject research. This practice demonstrates the ethical principle of "respect for persons." Our study was designed to imagine an informed consent future, specifically in a digital health context in which informed consent processes are mediated by sociotechnical systems. Design speed-dating workshops were conducted to explore dimensions of the consent communication design space, including social media, interactive quizzes, chat-bots, annotation tools, and virtual learning sessions. To explore both the user experience and how futuristic consent processes might be facilitated, the workshops involved people eligible to participate in digital health research (N=21) and service providers (N=20), including researchers and IRB members. Our findings offer five principles to improve digital informed consent processes: be concise, promote transparency, value time and effort, cultivate trust, and navigate platform risks.

How does presenting comments in a news article affect the ways that readers engage with and retain information about news? This paper presents results from a controlled experiment investigating effects related to different strategies for promoting discussion at news websites (N=336 participants). The strategies include highlighting specific comments about a data visualization, providing prompts with the comments, and annotating prompts on the visualization. By comparison to a simple list of comments (baseline), our analysis found that annotations contributed to higher levels of participant engagement in the discussion, yet lower levels of knowledge retention related to the article. These findings raise new considerations about whether and how to integrate discussion content into news and points toward future content moderation systems that assist in representing and eliciting discussion at news websites.

Due to challenges around low-quality comments and misinformation, many news outlets have opted to turn off commenting features on their websites. The New York Times (NYT), on the other hand, has continued to scale up its online discussion resources to reach large audiences. Through interviews with the NYT moderation team, we present examples of how moderators manage the first ~24 hours of online discussion after a story breaks, while balancing concerns about journalistic credibility. We discuss how managing comments at the NYT is not merely a matter of content regulation, but can involve reporting from the "community beat" to recognize emerging topics and synthesize the multiple perspectives in a discussion to promote community. We discuss how other news organizations---including those lacking moderation resources---might appropriate the strategies and decisions offered by the NYT. Future research should investigate strategies to share and update the information generated about topics in the news through the course of content moderation.

Writing is a common task for crowdsourcing researchers exploring complex and creative work. To better understand how we write with crowds, we conducted both a literature review of crowd-writing systems and structured interviews with designers of such systems. We argue that the cognitive process theory of writing described by Flower and Hayes (1981), originally proposed as a theory of how solo writers write, offers a useful analytic lens for examining the design of crowd-writing systems. This lens enabled us to identify system design challenges that are inherent to the process of writing as well as design challenges that are introduced by crowdsourcing. The findings present both similarities and differences between how solo writers write versus how we write with crowds. To conclude, we discuss how the research community might apply and transcend the cognitive process model to identify opportunities for future research in crowd-writing systems.

News websites can facilitate global discussions about civic issues, but the financial cost and burden of moderating these forums has forced many to disable their commenting systems. In this paper, we consider the role that data visualizations play in online discussion around a civic issue, through an analysis of how people talk about climate change data in the comment threads at three news websites (i.e., Breitbart news, the Guardian, the New York Times). We find that out of 6,525 comments, only 2.4% reference data visualizations in the articles. While rare, the paper presents illustrative examples of how people refer to data---their collection, analysis, and visual representation---to engage with an article's narrative. Using text classification techniques we identify several features related to the content of comments that contain data-centered talk, such as article cosine similarity, hyperlinks, and comparison terms. Finally, we discuss potential ways that newsrooms might apply this analysis to promote data literacy, data science, and to foster community around shared experiences.

Public concern related to a policy may span a range of topics. As a result, policy discussions struggle to deeply examine any one topic before moving to the next. In policy deliberation research, this is referred to as a problem of topical coherence. In an experiment, we curated the comments in a policy discussion to prioritize arguments for or against a policy proposal, and examined how this curation and participants' initial positions of support or opposition to the policy affected the coherence of their contributions to existing topics. We found an asymmetric interaction between participants' initial positions and comment curation: participants with different initial positions had unequal reactions to curation that foregrounded comments with which they disagreed. This asymmetry implies that the factors underlying coherence are more nuanced than prioritizing participants' agreement or disagreement. We discuss how this finding relates to curating for coherent disagreement, and for curation more generally in deliberative processes.

Inspired by policy deliberation methods and iterative writing in crowdsourcing, we developed and evaluated a task in which newcomers to an online policy discussion, before entering the discussion, generate prompts that encourage existing commenters to engage with each other. In an experiment with 453 Amazon Mechanical Turk (AMT) crowd workers, we found that newcomers can often craft acceptable prompts, especially when given guidance on prompt-writing and balanced opinions between the comments they synthesize. However, crafting these prompts had little effect on the quality of comments they posted to a simulated discussion forum following the prompt task, as measured by the reasoning and topic coherence of comments. Our results inform best practices and pose questions for the design of discussion systems, both in general and for online policy discussion in particular.

From Fortune 500 companies to local communities, organizations often strive to build a shared understanding about complex problems. Design competitions provide a compelling approach to create incentives and infrastructure for gathering insights about a problem-space. In this paper, we present an analysis of a two-month civic design competition focused on transportation challenges in a major US city. We examine how the event structure, discussion platform, and participant interactions affected how a community collectively discussed design constraints and proposals. Ninety-two participants took part in the competition's online discussion, hosted on Slack. Applying a mixed-methods analysis, we found that participants shared less as they settled into teams and, due to the discussion system, had difficulty seeing how topics connected across channels; we also learned that certain messages led participants to add depth to existing topics. Based on the findings we provide recommendations for civic competitions aimed at building knowledge around a problem.

Online crowd labor markets often address issues of risk and mistrust between employers and employees from the employers' perspective, but less often from that of employees. Based on 437 comments posted by crowd workers (Turkers) on the Amazon Mechanical Turk (AMT) participation agreement, we identified work rejection as a major risk that Turkers experience. Unfair rejections can result from poorly-designed tasks, unclear instructions, technical errors, and malicious Requesters. Because the AMT policy and platform provide little recourse to Turkers, they adopt strategies to minimize risk: avoiding new and known bad Requesters, sharing information with other Turkers, and choosing low-risk tasks. Through a series of ideas inspired by these findings-including notifying Turkers and Requesters of a broken task, returning rejected work to Turkers for repair, and providing collective dispute resolution mechanisms-we argue that making reducing risk and building trust a first-class design goal can lead to solutions that improve outcomes around rejected work for all parties in online labor markets.

Often, attention to “community” focuses on motivating core members or helping newcomers become regulars. However, much of the traffic to online communities comes from people who visit only briefly. We hypothesize that their personal characteristics, design elements of the site, and others' activity all affect the contributions these "one-timers" make. We present the results from an experiment asking Amazon Mechanical Turk (“AMT”) workers to comment on the AMT participation agreement in a discussion forum. One-timers with stronger ties to other Turkers or feelings of trust for Amazon are more likely to leave more --- but shorter and less relevant --- comments, while those with higher self-efficacy leave longer and more relevant comments. The phrasing of prompts also matters; a general appeal for personally-reflective contributions leads to comments that are less relevant to community discussion topics. Finally, activity matters too; synchronous activity begets responses, while pre-existing content tends to suppress them. These findings suggest design moves that can help communities harness this “long tail” of contribution.

The RAND Database of Worldwide Terrorism Incidents (RDWTI) seeks to index information about all terrorist incidents that occur and are mentioned in worldwide news media, providing a useful resource for policy researchers and decision makers. We examined automated classification methods that could be used to identify news articles about terrorist incidents, thus enabling analysts to read a smaller number of news articles and maintain the database with less effort and cost. The support vector machine (SVM) and Lasso methods were only modestly successful, but a classifier based on the gradient boosting method (GBM) appeared to be very successful, correctly ranking 80% of the relevant articles at the “top of the pile” for examination by a human analyst.