Artificial intelligence algorithms have been used to enhance a wide variety of products and services, including assisting human decision making in high-stake contexts. However, these algorithms are complex and have trade-offs, notably between prediction accuracy and fairness to population subgroups. This makes it hard for designers to understand algorithms and design products or services in a way that respects users' goals, values, and needs. We proposed a method to help designers and users explore algorithms, visualize their trade-offs, and select algorithms with trade-offs consistent with their goals and needs. We evaluated our method on the problem of predicting criminal defendants' likelihood to re-offend through (i) a large-scale Amazon Mechanical Turk experiment, and (ii) in-depth interviews with domain experts. Our evaluations show that our method can help designers and users of these systems better understand and navigate algorithmic trade-offs. This paper contributes a new way of providing designers the ability to understand and control the outcomes of algorithmic systems they are creating.

On Wikipedia, sophisticated algorithmic tools are used to assess the quality of edits and take corrective actions. However, algorithms can fail to solve the problems they were designed for if they conflict with the values of communities who use them. In this study, we take a Value-Sensitive Algorithm Design approach to understanding a community-created and -maintained machine learning-based algorithm called the Objective Revision Evaluation System (ORES)---a quality prediction system used in numerous Wikipedia applications and contexts. Five major values converged across stakeholder groups that ORES (and its dependent applications) should: (1) reduce the effort of community maintenance, (2) maintain human judgement as the final authority, (3) support differing peoples' differing workflows, (4) encourage positive engagement with diverse editor groups, and (5) establish trustworthiness of people and algorithms within the community. We reveal tensions between these values and discuss implications for future research to improve algorithms like ORES.

Group projects are an essential component of teaching user interface (UI) design. We identified six challenges in transferring traditional group projects into the context of Massive Open Online Courses: managing dropout, avoiding free-riding, appropriate scaffolding, cultural and time zone differences, and establishing common ground. We present a case study of the design of a group project for a UI Design MOOC, in which we implemented technical tools and social structures to cope with the above challenges. Based on survey analysis, interviews, and team chat data from the students over a six-month period, we found that our socio-technical design addressed many of the obstacles that MOOC learners encountered during remote collaboration. We translate our findings into design implications for better group learning experiences at scale.

Online platforms provide new channels for people in need to seek help from friends and strangers. However, individuals often encounter psychological barriers that deter them from asking for help. For example, people might have different concerns about asking for help, including acknowledging incompetence, bothering others, and accruing social debt. These perceived social costs limit the potential benefits of help solicitations. In this study, we attempt to investigate whether anonymity (posting a question anonymously), ephemerality (allowing questions to be visible for only a short period), and system routing (having the system handle the question routing) could reduce social costs in a typical online help-seeking behavior-question asking. We built a platform to support these three features and conducted a controlled within-subjects experiment to test their effects on the social costs of posting questions. Results suggest that the presence of anonymity, ephemerality, and system routing reduce social costs. Further, we find that employing anonymity and system routing features did not lower the quality and quantity of answers to the questions in our system.

Organizers of online groups often struggle to recruit members who can most effectively carry out the group's activities and remain part of the group over time. In a study of a sample of 30,000 new editors belonging to 1,054 English WikiProjects, we empirically examine the effects of generalized prior work-productivity experience (measured by overall prior article edits), prior leadership experience (measured by overall prior project edits), and localized prior work-productivity experience (measured by pre-joining article edits on a project) on early retention and productivity. We find that (1)generalized prior work-productivity experience is positively associated with retention, but negatively associated with productivity (2) prior leadership experience is negatively associated with both retention and productivity, and (3) localized prior work-productivity experience is positively associated with both retention and productivity within that focal project. We then discuss implications to inform the designs of early interventions aimed at group success.

Most commonly used approaches to developing automated or artificially intelligent algorithmic systems are Big Data-driven and machine learning-based. However, these approaches can fail, for two notable reasons: (1) they may lack critical engagement with users and other stakeholders; (2) they rely largely on historical human judgments, which do not capture and incorporate human insights into how the world can be improved in the future. We propose and describe a novel method for the design of such algorithms, which we call Value Sensitive Algorithm Design. Value Sensitive Algorithm Design incorporates stakeholders' tacit knowledge and explicit feedback in the early stages of algorithm creation. This increases the chance to avoid biases in design choices or to compromise key stakeholder values. Generally, we believe that algorithms should be designed to balance multiple stakeholders' needs, motivations, and interests, and to help achieve important collective goals. We also describe a specific project "Designing Intelligent Socialization Algorithms for WikiProjects in Wikipedia" to illustrate our method. We intend this paper to contribute to the rich ongoing conversation concerning the use of algorithms in supporting critical decision-making in society.

Although we may complain about meetings, they are an essential part of an information worker's work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant executing an unstructured macrotask. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.

Wikipedia-based studies and systems frequently assume that no two articles describe the same concept. However, in this paper, we show that this article-as-concept assumption is problematic due to editors' tendency to split articles into parent articles and sub-articles when articles get too long for readers (e.g. "Portland, Oregon" and "History of Portland, Oregon" in the English Wikipedia). In this paper, we present evidence that this issue can have significant impacts on Wikipedia-based studies and systems and introduce the sub-article matching problem. The goal of the sub-article matching problem is to automatically connect sub-articles to parent articles to help Wikipedia-based studies and systems retrieve complete information about a concept. We then describe the first system to address the sub-article matching problem. We show that, using a diverse feature set and standard machine learning techniques, our system can achieve good performance on most of our ground truth datasets, significantly outperforming baseline approaches.

Productive and dedicated members are critical to the success of online production communities like Wikipedia. Many communities organize in subgroups where members voluntarily work on projects of shared interest. In this paper, we investigate how members' pre-joining connections with the subgroup predict their productivity and withdrawal after joining. Drawing insights from attachment theories in social psychology, we examine two types of pre-joining connections: textit{identity-based} attachment (how much members' interests were aligned with the subgroup's topics) and textit{bonds-based} attachment (how much members had interacted with other members of the subgroup). Analyses of 79,704 editors in 1,341 WikiProjects show that 1) both identity-based and bonds-based attachment increased editors' post-joining productivity and reduced their likelihood of withdrawal; 2) identity-based attachment had a stronger effect on boosting direct contributions to articles while bonds-based attachment had a stronger effect on increasing article and project coordination, and reducing member withdrawal.

Team project, which emphasizes collaborative learning in a project-based context, is one of the most commonly-used teaching and learning methods in higher education classrooms, but is not well-supported on existing Massive Open Online Course (MOOC) platforms. In this paper, we present ProjectLens, a MOOC supplement tool that supports team projects building and collaborative learning on MOOC platforms like Coursera and edX. In addition, ProjectLens is a research tool that provides opportunities to conduct large-scale field experiments to study how different factors influence the effectiveness of collaborative learning. We illustrate how ProjectLens can achieve these two goals in a case example.

Nearly any group is subject to turnover : some people leave, while others join. Turnover can be especially high in online groups, since participation typically is strictly voluntary. We investigated the effects of member turnover in online groups, specifically in Wikipedia's WikiProjects. We based our studies on theories from organizational science, which suggest that it is not just the amount of turnover, but the characteristics of those leaving and those joining that matter. We characterized leavers and newcomers by their prior productivity, tenure (in the group or community), and participation in other groups within the larger community. Furthermore, we considered the moderating effect of group size on turnover. We analyzed data from 88,427 editors who participated in 1,054 WikiProjects, finding that (1) the positive effects of newcomers to a group were larger than the negative effects of leavers, (2) prior productivity, tenure, and participation in other groups all played significant roles, and (3) the effects of leavers and newcomers were amplified in larger groups.