In response to calls for open data and growing privacy threats, organizations are increasingly adopting privacy-preserving techniques that add noise to published datasets. These techniques seek to protect privacy of data subjects while enabling useful analyses. With expert feedback, we developed empirically-driven documentation explaining the noise characteristics of two Wikipedia pageview datasets: one using rounding (heuristic privacy) and another using differential privacy (DP, formal privacy). We then used these documents to conduct a task-based contextual inquiry (n=15) exploring how data users—largely unfamiliar with these methods—perceive, interact with, and interpret privacy-preserving noise during data analysis.

We present a new user-centered design course that prepares computing undergraduates for software industry jobs such as UI/UX designer, product designer, and product manager. Our course aims to bridge the academia-industry gap and innovates upon prior published HCI courses due to its targeted focus on job preparation, inclusion, and scale. Nearly 200 students (55% women) have taken it in the past two years. We developed its curriculum to align with the needs of modern industry employers and implemented five theory-backed pedagogical principles: 1) industry-relevant project prompts developed in consultation with recent course alumni, 2) final project deliverable optimized for job-seeking, 3) no coding required to foster inclusion, 4) low-stress effort-based grading to further foster inclusion, 5) weekly feedback and chances for revisions. We discuss the theoretical rationale behind these five principles and how instructors can potentially apply them to a broad range of project-based courses across many areas of computing.

Plots and tables are commonplace in today’s data-driven world, and much research has been done on how to make these figures easy to read and understand. Often times, however, the information they contain conveys only the end result of a complex and subtle data analysis pipeline. This can leave the reader struggling to understand what steps were taken to arrive at a figure, and what implications this has for the underlying results. In this paper, we introduce datamations, which are animations designed to explain the steps that led to a given plot or table. We present the motivation and concept behind datamations, discuss how to programmatically generate them, and provide the results of two large-scale randomized experiments investigating how datamations affect people’s abilities to understand potentially puzzling results compared to seeing only final plots and tables containing those results.

Data scientists often collaborate with clients to analyze data to meet a client's needs. What does the end-to-end workflow of a data scientist's collaboration with clients look like throughout the lifetime of a project? To investigate this question, we interviewed ten data scientists (5 female, 4 male, 1 non-binary) in diverse roles across industry and academia. We discovered that they work with clients in a six-stage outer-loop workflow, which involves 1) laying groundwork by building trust before a project begins, 2) orienting to the constraints of the client's environment, 3) collaboratively framing the problem, 4) bridging the gap between data science and domain expertise, 5) the inner loop of technical data analysis work, 6) counseling to help clients emotionally cope with analysis results. This novel outer-loop workflow contributes to CSCW by expanding the notion of what collaboration means in data science beyond the widely-known inner-loop technical workflow stages of acquiring, cleaning, analyzing, modeling, and visualizing data. We conclude by discussing the implications of our findings for data science education, parallels to design work, and unmet needs for tool development.

Hundreds of millions of people learn something new online every day. Simultaneously, the study of online education has blossomed within the human computer interaction community, with new systems, experiments, and observations creating and exploring previously undiscovered online learning environments. In this study we endeavor to characterize this entire landscape of online learning experiences using a national survey of 2260 US adults who are balanced to match the demographics of the U.S. We examine the online learning resources that they consult, and we analyze the subjects that they pursue using those resources. Furthermore, we compare both formal and informal online learning experiences on a larger scale than has ever been done before, to our knowledge, to better understand which subjects people are seeking for intensive study. We find that there is a core set of online learning experiences that are central to other experiences and these are shared among the majority of people who learn online. We conclude by showing how looking outside of these core online learning experiences can reveal opportunities for innovation in online education.

Data science has been growing in prominence across both academia and industry, but there is still little formal consensus about how to teach it. Many people who currently teach data science are practitioners such as computational researchers in academia or data scientists in industry. To understand how these practitioner-instructors pass their knowledge onto novices and how that contrasts with teaching more traditional forms of programming, we interviewed 20 data scientists who teach in settings ranging from small-group workshops to large online courses. We found that: 1) they must empathize with a diverse array of student backgrounds and expectations, 2) they teach technical workflows that integrate authentic practices surrounding code, data, and communication, 3) they face challenges involving authenticity versus abstraction in software setup, finding and curating pedagogically-relevant datasets, and acclimating students to live with uncertainty in data analysis. These findings can point the way toward better tools for data science education and help bring data literacy to more people around the world.

Elissa Redmiles, Lisa Maszkiewicz, Emily Hwang, Dhruv Kuchhal, Everest Liu, Miraida Morales, Denis Peskov, Sudha Rao, Rock Stevens, Kristina Gligorić, Sean Kross, Michelle Mazurek, Hal Daumé III. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Security and privacy researchers often rely on data collected from Amazon Mechanical Turk (MTurk) to evaluate security tools, to understand users' privacy preferences and to measure online behavior. Yet, little is known about how well Turkers' survey responses and performance on security- and privacy-related tasks generalizes to a broader population. This paper takes a first step toward understanding the generalizability of security and privacy user studies by comparing users' self-reports of their security and privacy knowledge, past experiences, advice sources, and behavior across samples collected using MTurk (n=480), a census-representative web-panel (n=428), and a probabilistic telephone sample (n=3,000) statistically weighted to be accurate within 2.7% of the true prevalence in the U.S. Surprisingly, the results suggest that: (1) MTurk responses regarding security and privacy experiences, advice sources, and knowledge are more representative of the U.S. population than are responses from the census-representative panel; (2) MTurk and general population reports of security and privacy experiences, knowledge, and advice sources are quite similar for respondents who are younger than 50 or who have some college education; and (3) respondents' answers to the survey questions we ask are stable over time and robust to relevant, broadly-reported news events. Further, differences in responses cannot be ameliorated with simple demographic weighting, possibly because MTurk and panel participants have more internet experience compared to their demographic peers. Together, these findings lend tempered support for the generalizability of prior crowdsourced security and privacy user studies; provide context to more accurately interpret the results of such studies; and suggest rich directions for future work to mitigate experience- rather than demographic-related sample biases.

Efforts to improve diversity in computing have mostly focused on K-12 and university student populations, so there is a lack of research on how to provide these benefits to adults who are not in school. To address this knowledge gap, we present a case study of how a nine-member team of end-user programmers designed an educational program to bring job-relevant computing skills to adult populations that have traditionally not been reached by existing efforts. This team conceived, implemented, and delivered Cloud Based Data Science (CBDS), a data science course designed for adults in their local community in historically marginalized groups that are underrepresented in computing fields. Notably, nobody on the course development team was a full-time educator or software engineer. To reduce the amount of time and cost required to launch their program, they repurposed end-user programming skills and tools from their professions, such as data-analytic programming and reproducible scientific research workflows. This case study demonstrates how the spirit of end-user programming can be a vehicle to drive social change through grassroots efforts.

The security field relies on user studies, often including survey questions, to query end users' general security behavior and experiences, or hypothetical responses to new messages or tools. Self-report data has many benefits -- ease of collection, control, and depth of understanding -- but also many well-known biases stemming from people's difficulty remembering prior events or predicting how they might behave, as well as their tendency to shape their answers to a perceived audience. Prior work in fields like public health has focused on measuring these biases and developing effective mitigations; however, there is limited evidence as to whether and how these biases and mitigations apply specifically in a computer-security context. In this work, we systematically compare real-world measurement data to survey results, focusing on an exemplar, well-studied security behavior: software updating. We align field measurements about specific software updates (n=517,932) with survey results in which participants respond to the update messages that were used when those versions were released (n=2,092). This allows us to examine differences in self-reported and observed update speeds, as well as examining self-reported responses to particular message features that may correlate with these results. The results indicate that for the most part, self-reported data varies consistently and systematically with measured data. However, this systematic relationship breaks down when survey respondents are required to notice and act on minor details of experimental manipulations. Our results suggest that many insights from self-report security data can, when used with care, translate to real-world environments; however, insights about specific variations in message texts or other details may be more difficult to assess with surveys.

We survey all four years of papers published so far at the Learning at Scale conference in order to reflect on the major research areas that have been investigated and to chart possible directions for future study. We classified all 69 full papers so far into three categories: Systems for Learning at Scale, Interactions with Sociotechnical Systems, and Understanding Online Students. Systems papers presented technologies that varied by how much they amplify human effort (e.g., one-to-one, one-to-many, many-to-many). Interaction papers studied both individual and group interactions with learning technologies. Finally, student-centric study papers focused on modeling knowledge and on promoting global access and equity. We conclude by charting future research directions related to topics such as going beyond the MOOC hype cycle, axes of scale for systems, more immersive course experiences, learning on mobile devices, diversity in student personas, students as co-creators, and fostering better social connections amongst students.

The behavior of the least-secure user can influence security and privacy outcomes for everyone else. Thus, it is important to understand the factors that influence the security and privacy of a broad variety of people. Prior work has suggested that users with differing socioeconomic status (SES) may behave differently; however, no research has examined how SES, advice sources, and resources relate to the security and privacy incidents users report. To address this question, we analyze a 3,000 respondent, census-representative telephone survey. We find that, contrary to prior assumptions, people with lower educational attainment report equal or fewer incidents as more educated people, and that users' experiences are significantly correlated with their advice sources, regardless of SES or resources.

Few users have a single, authoritative, source from whom they can request digital-security advice. Rather, digital-security skills are often learned haphazardly, as users filter through an overwhelming quantity of security advice. By understanding the factors that contribute to users' advice sources, beliefs, and security behaviors, we can help to pare down the quantity and improve the quality of advice provided to users, streamlining the process of learning key behaviors. This paper rigorously investigates how users' security beliefs, knowledge, and demographics correlate with their sources of security advice, and how all these factors influence security behaviors. Using a carefully pre-tested, U.S.-census-representative survey of 526 users, we present an overview of the prevalence of respondents' advice sources, reasons for accepting and rejecting advice from those sources, and the impact of these sources and demographic factors on security behavior. We find evidence of a "digital divide" in security: the advice sources of users with higher skill levels and socioeconomic status differ from those with fewer resources. This digital security divide may add to the vulnerability of already disadvantaged users. Additionally, we confirm and extend results from prior small-sample studies about why users accept certain digital-security advice (e.g., because they trust the source rather than the content) and reject other advice (e.g., because it is inconvenient and because it contains too much marketing material). We conclude with recommendations for combating the digital divide and improving the efficacy of digital-security advice.

In 2012, women earned 18% of computer science degrees; African American and Hispanic students made up less than 20% of computing degree holders that year. Research shows that relatable role models and engaging curriculum are required to engage underrepresented students in computing. There is a need for engaging and relatable curriculum to be delivered to students at the middle school level, when these students first begin to lose interest in computing. Thus, based on the results of a survey of current and former middle school computing teachers and a comprehensive literature review, we developed the University of Maryland Curriculum In A Box (CIAB). The CIAB includes profiles of relatable computing role models, accessible video and text curriculum and challenge projects for HTML/CSS. To simulate a "real world" programming environment, the CIAB guides students through programming within open source social media frameworks and Github. The CIAB also includes teacher enablement resources such as assessments and a week-by-week implementation guide. The CIAB was successfully implemented with a group of 6th and 7th grade students in Prince Georges (PG) County, a majority minority county in Maryland. Our demo will provide a walk-through of the CIAB assets, accessibility features and design process, as well as implementation advice informed by our CIAB implementation in PG County.