The student journey in degree programs can be viewed as a pipeline: applicants must be admitted, admitted students must matriculate, and matriculants must persist and perform. Using administrative records from over 50,000 applicants to an online graduate computing program, we model these three stages as distinct outcomes: admission, matriculation among admitted students, and longer-run success among matriculants. For each stage, we estimate multivariable logistic models using structured application attributes (e.g., age, citizenship, application history, undergraduate GPA) and highdimensional text features from self-reported academic and employment histories (TF-IDF). We then compare success models based on application data alone versus application data augmented with firstterm performance and enrollment patterns. Across stages, models reveal different correlates of advancement through the pipeline: application features strongly differentiate admission and yield, while long-run success is only weakly explained at application time but becomes highly explainable when we include first-term academic momentum. Together, the results clarify what can be inferred at each decision point and highlight the outsized role of early-course performance for retention-oriented interventions.

At the time of data collection in Fall 2024, our Online Master of Science in Computer Science program in the College of Computing at Georgia Tech had its largest applicant pool to date. Leveraging this large sample, we surveyed all 6,247 applicants to understand their motivations for applying to our program. This paper focuses specifically on the open-ended responses from female applicants, with the goal of informing outreach and recruitment strategies that support greater female inclusion in computer science graduate education. The analysis of open-ended responses revealed insights into the importance of career advancement, social networks, and the program's courses as motivations for female applicants.

The flexibility and affordability of online, asynchronous, at-scale degree programs have significantly increased the accessibility of a master's-level graduate education. While studies have been conducted on the general growth of such programs and the quality of the online courses compared to their on-campus counterparts, few (if any) have examined outcomes such as alumni career growth or admission into other graduate programs. This work examines how one large online graduate program in computer science prepared alumni for matriculation into STEM PhD programs. Enrollment data from the National Student Clearinghouse was analyzed to identify key trends in alumni PhD enrollment. Surveys and interviews with program alumni were also conducted to investigate the unique paths that these individuals took to beginning their PhD education. This study finds that the program positively impacted alumni PhD experiences in STEM fields. Alumni noted that involvement with graduate research and coursework were key components in their preparation for a PhD program. These results demonstrate that an affordable, online, asynchronous graduate STEM program can provide non-traditional students with an effective pathway to PhD enrollment. The paper concludes with recommendations for asynchronous, at-scale degree programs seeking to expand their research opportunities for students with a desire to pursue PhD programs.

This study analyzes patterns of responses from a series of student surveys that predict student success in a CS1 course at a major research university in the United States. The research identifies statistically significant effects on final grades from self-reported course interaction perceptions and preferences, providing insights into designing future studies and potential early interventions.

Growing demand for online education in recent years has had a profound impact on how academic institutions educate students, resulting in a massive increase in online at-scale curricula. As on campus higher education students interact with online content, there is a need to examine how such students have adjusted in response. In this research, we investigate the impact of integrating online at-scale content into more traditional educational environments through two studies examining the learner's perspective. In the first study, we surveyed 22 undergraduate and graduate students who were enrolled in a course that leveraged online at-scale content in the course's delivery. In the second study, we conducted interviews with 17 undergraduate and graduate students in similar situations for a more in-depth understanding of their experiences. From these two studies, we then discuss key themes, including both benefits and drawbacks of these environments, that students derived from their course experiences.

Gateway courses play a key role in student success.These courses often control access to advanced coursework and can delay graduation if students struggle to pass them.Identifying such courses is important for curriculum design and student support.However, existing methods rely on heuristics or past grades and often overlook complex patterns in student enrollment data.In this paper, we propose a tensor decomposition-based method to identify gateway courses in a systematic way.We model studentcourse-semester interactions as a three-dimensional tensor.This captures relationships between students, courses, and time.Using PARAFAC tensor decomposition, we extract latent factors that represent course importance based on their structure in the data.Our analysis shows that this method highlights courses that are central, or distinctive in the curriculum.Courses with high latent factor magnitudes are likely to represent gateway courses or critical prerequisites.As future work, we plan to combine tensor decomposition with learning techniques.This could enable predictive models and provide insights for curriculum design.Our goal is to develop a complete framework to analyze educational data, and support institutions in improving student outcomes.

After several years of relatively stable application growth, one online at-scale CS graduate program experienced a major uptick in applications in 2024. Like other at-scale degree programs, this program generally accepts all qualified students, and the number of matriculating students is projected based on applications rather than established based on capacity. As a result, understanding drivers of application growth is important to anticipate unexpected sudden increases in demand. In this analysis, we chart trends in application growth over this program's history and report the results of a survey sent to the application class of 2024 to better understand their motivations for applying.

Instructors of large online courses often rely on asynchronous online discussions (AOD) to engage students, build community, and enhance collaborative learning. Despite these documented benefits, AOD frequently encounter challenges such as superficial interactions, uneven participation, and motivational barriers. In contrast, peer assessment---an engagement strategy involving structured peer feedback---has been shown to promote deep cognitive engagement. Despite their complementary potential, few studies have directly compared peer assessment and AOD as distinct, coexisting engagement strategies within the same learning environment. This study addresses this critical gap by examining the relative impacts of AOD and peer assessment on student learning performance and perceptions in a large online graduate course in computer science (N = 1,451) over five years (Spring 2020--Fall 2024). Students were classified based on participation: Collaborators (active in both activities), Reviewers (active primarily in peer assessment), Discussants (active primarily in online discussions), and Limited Contributors (limited participation). ANOVA results showed Collaborators and Reviewers significantly outperformed Discussants and Limited Contributors, indicating that peer assessment is strongly linked to improved performance. Additionally, Discussants outperformed Limited Contributors, reaffirming the engagement value of AOD. Regression analysis further revealed that peer assessment was a stronger predictor than AOD for both perceived peer support and course effectiveness, with a larger predictive gap observed for peer support. These findings highlight the complementary potential of combining peer assessment with AOD in enhancing engagement and performance in large-scale online learning environments. Educators are encouraged to strategically combine these strategies to leverage their distinct strengths, while future research should explore interventions to better engage minimally active students.

Artificial intelligence (AI) is increasingly integrated into project-based learning (PjBL), with research primarily addressing two dimensions: learning about AI through PjBL and learning with AI as a supportive tool. However, studies on AI's role in PjBL remain limited, often focusing on small-scale, short-term contexts. Additionally, there is little focus on a third dimension---learning through AI Creation---where students deepen engagement by developing AI-driven projects. This study investigates AI's dual role in online PjBL at scale, analyzing how students use AI tools and engage in AI-driven projects within a graduate computer science course over six semesters (2023-2024). Using a mixed-methods approach, we analyzed survey responses from 467 students and conducted a thematic analysis of 436 unique project keyword sets. Findings reveal that students primarily use AI for technical tasks, research, and content generation, supporting PjBL's planning and execution phases. The analysis of students' project keywords highlights AI in Education as the most prevalent project theme, with subthemes like Large Language Models, Intelligent Tutoring Systems, and Personalized Learning, indicating strong student interest in leveraging AI to address educational challenges related to personalization and scalability. The diversity of AI subthemes, including AI Ethics and AI for Accessibility, suggests that students are exploring AI creation through multiple perspectives, considering both technical and societal implications. This study contributes to the growing body of research on AI in education by offering large-scale, longitudinal insights into AI's dual role in online PjBL: as a tool for enhancing learning and as a medium for deeper engagement through AI creation. The findings have important implications for curriculum design, such as scaffolded AI creation and ethical training, preparing students for a future in which they are not mere consumers of AI technologies but also responsible AI innovators.

The Internet---for all of its benefits---makes it easy for students to share assignments. This creates a serious problem for academic institutions. Common mitigation tactics include discouraging students from sharing their work and routinely checking for and removing solutions shared online. While these strategies can be successful in many cases, they are not always sufficient. In our experience, it can be a challenge if either students or hosting sites refuse to remove solutions. Pursuing legal options can be both time consuming and costly. One approach taken to combat this is to routinely create new coding assignments, but this can still require a significant time commitment. It is worth exploring if this effort is worthwhile.

International research collaboration (IRC) leads to stronger, more visible research, and it allows researchers to account for global perspectives in their work. This is particularly important in Learning @ Scale research, as "scale" often implies learning that spans countries or continents. This paper examines IRC at Learning @ Scale by analyzing the authorship of all Learning @ Scale papers over the conference's 10-year history. We find that IRC at the conference is low, and possibly trending downwards. Additionally, authors from South American, African, and Asian countries tend to favor IRC, and those countries that publish most frequently at Learning @ Scale tend to be less internationally collaborative. This paper serves as a call to action for researchers and the conference to increase IRC, and we provide recommendations for accomplishing this goal.

This paper examines research trends from the first 10 years of the Learning @ Scale conference across three dimensions: research context, subject matter, and the countries of the authors' affiliations. Each of the 562 papers published over the conference's history was coded across these dimensions. The results reveal a significant drop in MOOC research, a rise in research from outside the United States, and a consistent focus on computer science and STEM curricula.

Ten years ago in 2014 saw the launch of the first in what would become a trend to launch online, at-scale degree programs (mostly at the graduate level) by leveraging MOOC pedagogies and platforms. Using publicly-available data, this paper describes ten trends that characterize the first decade of one such program in computer science in order to assess the audience and performance of students who elect to enroll in such programs. Past research has found that students in these programs tend to enroll in large part because they do not have other options for rigorous, respected credentials that fit into their professional and personal lives; this study unpacks many of these trends. Specifically, this study finds that the fraction of women enrolling in the program has steadily increased over time, that the gender discrepancy can be partially explained by underlying differences in women in CS across different nationalities, that applicants to the program tend to be evenly split between technical and non-technical backgrounds, and that the geographic distribution of students over time has shifted toward more international audiences even while the state in which the program originates comprises the largest fraction per capita of enrollees. The paper concludes by discussing how these trends might generalize to feedback to other similar programs and other at-scale initiatives.

Research consistently shows that peer assessment affects student achievement and attitudes across various subjects and contexts. However, most studies have focused on anonymous peer assessments in small, one-time, and non-iterative settings. This large-scale longitudinal study explores the effects of open, collaborative, and AI-augmented peer assessment in a large online graduate course with 1,636 students across 12 semesters from 2018 to 2022. The research investigated how different groups of students participated in the peer assessment in an online graduate computer science class. We also explored how their participation influenced their learning performance and their perceptions. Key findings reveal that students' age and gender significantly affect engagement levels and the perception of peer assessment effectiveness. They also provide new insights into the positive relationship between providing feedback and enhanced learning performance. This paper presents the implementation of peer assessment and detailed findings of the study. The implications of the study for future research and practices are also discussed.

This work in progress paper examines the impact of LLMs such as ChatGPT in a college-level introductory computing course offered simultaneously as a massive open online course (MOOC) on the edX platform, focusing on its strengths and limitations in solving coding assignments. The study reveals ChatGPT's proficiency in some areas while highlighting challenges in pseudo-code interpretation, handling multiple correct answers, and addressing complex problem statements. In order to discourage over-reliance on AI assistance from students while preserving scalability, the paper proposes strategies to enhance the difficulty of coding assignments by adding more creative elements in their structure. This research provides insights into the dynamics of AI in education and emphasizes the need for a balanced approach between technological assistance and genuine student participation.

The emerging field of affordable degrees at scale relies in large part on a linear relationship between the number of students who enroll and the number of teaching assistants who are employed. As programs grow, however, identifying good candidates can become untenable: research has found that the fraction of students who apply for TA roles far exceeds the number of actual available roles, forcing instructors to comb through hundreds of applications for a tiny number of positions. Most of these applicants are themselves former students in the class as well, meaning that there is abundant data from their assignments, grades, peer review behaviors, and forum participation to evaluate candidates, but navigating and using this data requires significant time.

Research on how students prepare for graduate computer science programs typically focuses on single, subject-specific interventions or relates to preparation for life as a graduate student. Preparatory work completed prior to enrolling in such a program can be particularly important for underrepresented minorities and those without technical backgrounds. We use survey data from incoming students in a large online graduate computer science program to answer three research questions: What are the backgrounds of students entering the program? How do students prepare for the program? And how does student preparation differ based on demographics and prior experience? We find that: male students are more likely than female students to enter the program with computer science qualifications; older students, female students, and those with non-technical degrees are more likely to pursue preparation; and students with no online learning experience are less likely to pursue preparation. These findings highlight the importance of student backgrounds when creating preparatory courses and indicate the value of preparatory courses in increasing diversity in large online graduate programs.

We examine trends in the Learning at Scale conference from 2014 through 2021. We use an original coding scheme to classify all 142 full papers from five angles: setting, approach, pedagogical strategy, population of interest, and dependent variable. We observe a decline of research on MOOCs, an increase in number of settings studied over time, and a consistent focus on assessment strategies. We then examine other conferences to which Learning at Scale authors contribute research. This paper contributes an original coding scheme to classify future Learning at Scale papers; an analysis of the research focuses at Learning at Scale over time; and an evaluation of the mutual influence between Learning at Scale and other venues. These latter two contributions contextualize learning at scale research within Learning at Scale and the broader research community to show how the field has evolved and to help predict its future directions.

A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups-a control group and a treatment group. While the benefits of using A/B experiments are widely known and accepted in education, there is less agreement on an approach to creating software infrastructure systems to assist in rapidly conducting such experiments in the field. To assist in alleviating this gap, we are creating a software infrastructure for A/B experiments that allows researchers to conduct experiments and automatically analyze their results for an education-focused ecology-based conceptual modeling platform.

Proponents of at-scale education initiatives often tout the benefits of these programs to the students who enroll in them, but anecdotally, faculty who participate in creating at-scale courses have often commented on how their participation positively impacted their subsequent in-person teaching. In this study, we investigate this potential phenomenon more thoroughly to assess how generalizable this perspective is among participants in these initiatives. We conduct three studies: we interview 78 faculty teaching in at-scale degree programs, examine offering histories for 70 courses offered in one such program, and survey 153 faculty who have developed massive open online courses. Based on these three studies, we propose a list of ways in which in-person instruction benefits from the existence of at-scale teaching initiatives. We further discuss remaining perceived drawbacks of at-scale education as derived from these interviews and surveys.

Online education is convenient and accessible, but also brings new challenges to retention. Prior research has found that students in online programs---including CS programs---are more likely to withdraw both from individual classes as well as the program as a whole than students in traditional programs. In this study, we delve deeper into retention at the level of individual courses. We analyze three courses offered as part of a large online CS graduate degree program taught at a major research university in the United States. We obtained voluntary data from students across 12 semesters---including gender, prior CS experience, and anticipated workload for the course---and analyzed these variables in the context of course retention. While the average course drop rate for the examined courses in the program was 7%, variations in this rate could be predicted by a number of different attributes: for instance, we observed that women, native English speakers, and students with less prior education were allmore likely to complete a course that they had started. Importantly, these predictors can be measured before the class begins, supporting early intervention.

Using a Computer Science affordable degree at scale as a case study, the context of this paper explores the ways in which student life initiatives can humanize the at scale and online distance learning student experience through a holistic and student care-centric approach. By analyzing the top ten student life initiatives featured in an online at scale computer science program, we highlight ways to ensure belonging, facilitate engagement, and create community at scale while tackling inherent problems of online distance learning such as isolation and social disconnectedness.

Past research on the ways that students evaluate their courses has focused largely on how those evaluations relate to the specific course instructor. This research examines a set of data from a public site where students unofficially rate the courses in a very large online graduate program operating at scale. We examine the relationship between the unofficial scores students give to their classes with data on enrollment trends over time and the assessment strategies used within the courses themselves to examine additional actors that shape the ratings students choose, as well as how they use those ratings to choose what courses to take in the future. We find several different notable relationships: reviews in this context are largely impervious to the extreme response bias prevalent on other review sites; review content does not appear to significantly influence enrollment trends; more difficult classes tend to receive more favorable ratings overall, although individual students do not rate difficult classes more favorably; and project-based classes are perceived by students to be less difficult.

Students have developed their own platforms for sharing their evaluations of courses over the Internet. These are typically focused on the needs and experience of students in traditional undergraduate programs, but the rise of online programs operating at scale has made it practical for students to develop such a platform dedicated to their particular program. We have used a survey to gather information from students in such a program at a major research institution in the United States. Through this data we explore how many students are using the site, how they use the information, and also how often and why they write reviews. The ultimate goal is to gather information that could help students to decide how to critically assess such reviews and successfully use them to make better decisions.

"MOOC-based" degrees are degree programs often offered in partnership with MOOC providers that provide the flexibility and scale of MOOCs while also awarding accredited degrees. This positioning between MOOCs and degrees raises interesting questions regarding retention: MOOCs are famous for their low completion rates, but accredited degree programs often strive for high retention rates. This paper aims to answer the broad question: what does retention look like in a "MOOC-based" degree program? To answer this question, we analyze retention at two levels: first at the program level, then at the course level. We find that retention is far higher than in MOOCs, but notably lower than in traditional in-person programs, both when looking at the program as a whole and at individual courses. We provide discuss several hypotheses for this phenomenon, as well as implications for program evaluation and course design.

The term "double-edged sword" refers to something that may have both favorable and unfavorable consequences. We posit that allowing students to post anonymously in a CS course forum may fit this metaphor with regard to gender and belongingness. In this work, we test a theory that patterns of anonymous posting in a course forum for a CS1 class may reinforce gender stereotypes even as the underlying patterns of interaction debunk those stereotypes. We examine forum interactions from a CS1 class with an even gender split and find that women engage in anonymous posting more often than men; thus, a student's view of the class's gender distribution is different from the actual distribution. We hypothesize this is a missed opportunity to combat stereotypes of gender in computer science. Possible solutions and further work are discussed.

Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

The General Data Protection Regulation (GDPR), passed in 2018, outlined new data privacy standards applying to residents of the European Union (EU). The impact of this law stretches beyond the EU to anyone - such as researchers across the world - collecting or processing data from EU residents. Researchers have had to augment their methodologies to ensure GDPR compliance. This initiative intersects with at-scale educational programs, which often enroll EU students and are also often the subject of institutional research. While creating a study to research students in an online Master of Science in Computer Science (MSCS) program, some of whom are in the EU, we encountered difficulties with ensuring GDPR compliance. This paper discusses the implications of the GDPR related to both general research and our specific study. We discuss the challenges of interpreting the GDPR and integrating it into our methodology as well as potential solutions, our ultimate resolution, and practical recommendations, and we consider what the future of data privacy legislation means for researchers.

One of the major criticisms of efforts towards offering education at scale has been the Trap of Routine Assessment, the risk that student assessment will suffer from becoming excessively simplified in service of automation and scale. In this research, we examine the ways that students in an at-scale graduate program in computer science were assessed during their degrees. The program in question has scaled to over 10,000 students in only a few years, but awards a traditional Master's degree, providing the opportunity to investigate whether scale was achieved by transitioning to more routine assessment or by bringing scale to traditional strategies. To do this, we investigate the syllabi of 52 classes offered through the program to identify the types of assessments used, and we survey teaching teams for their approaches to evaluating these assessments. We merge this data with historical enrollment data to gain an overall summary of the kinds of assessments and evaluations received during their degrees. We ultimately find the program's scale has been managed by scaling up traditional assessment and evaluation strategies as the majority of grades are generated by human teaching teams based on projects and homeworks, with a relatively smaller portion generated exclusively by automated evaluation of exams.

Ensuring that students are fully informed about course content and policies is always a challenge, but online education at scale adds additional complications. In this paper we present observations about the place of the syllabus in education at scale, based on the actual syllabus documents from 48 courses in a Computer Science Master's degree program offered online and at scale. On the basis of these observations, we offer preliminary recommendations for factors that instructors should keep in mind when they compile a syllabus for similar courses.

Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one's home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.

In this research, we investigate the trajectory of attitudinal change towards computer science among students in an online CS1 class. We perform this investigation to address several trends in modern computer science education. First, as computer science increasingly becomes a required class for all majors, how do students' first experiences with the subject impact their attitudes? Second, as online education continues to expand, how does enrolling in CS1 online specifically affect audiences that may be marginalized in both CS classes and in online learning environments, such as women and underrepresented minorities? Third, can we intervene to improve attitudes towards computer science, especially among those marginalized audiences? In this research, we poll students in an online for-credit CS1 class four times to observe the change in their attitudes towards computer science over time and intervene with some students to try to improve their perception of computer science. We find that attitudes towards computer science improve with initial exposure, that women's attitudes towards CS begin less positive but follow the same trajectory, and that mid-semester regression in attitudes toward computer science may predict eventual struggles to perform well in the class.

Education has traditionally been administered via physical interactions between teachers and students in classrooms. Through technological advancement in communications and digital devices, online education has been developed with the potential to scale education, making it affordable and accessible. With an internet connection and a laptop or mobile phone, learners can access massive open online courses (MOOCs) for free. Nonetheless, the opportunity to scale education and the advantages of online learning are not always fulfilled due to certain challenges. In this work, Socioeconomic, Sociocultural, and IT infrastructural factors are categorized as challenges hindering the adoption of online learning in Nigeria. Although some factors mitigating online learning have been identified in the past, there is relatively little empirical evidence indicating the reality and severity of these challenges. Since scaling education involves worldwide reach, local contexts such as found in Nigeria and other developing countries become critical. The objective of this work, therefore, is to understand these challenges, present empirical evidence through a questionnaire survey, rank these challenges in order of severity, and propose solutions.

The goal of this workshop is to bring together the existing community of researchers working on Infrastructure Design for Data-Intensive Research in Computer Science Education and a community of Learning at Scale researchers focused on Computer Science Education. While both communities share many similar goals and could greatly benefit from each other work, the interaction between the communities is small. We hope that the proposed workshop will be instrumental in bringing together like-minded researchers from different communities, establishing collaboration, and expanding the scope of infrastructure project to address critical scaling issues.

While the literature on learning at scale has largely focused on MOOCs, online degree programs, and AI techniques for supporting scalable learning experiences, informal learning communities have been relatively underrepresented. None-theless, these massive open online learning communities regularly draw far more engaged users than the typical MOOC. Their informal structure, however, makes them significantly more difficult to study. In this work, we take a first step toward attempting to understand these communi-ties specifically from the perspective of scale. Taking a sample of 62 such communities, we develop a tagging sys-tem for understanding the specific features and how they relate to scale. For example, just as a MOOC cannot man-ually grade every assignment, so also an informal learning community cannot approve every contribution; and just as MOOCs therefore employ autograding, informal learning communities employ crowd-sourced moderation or plat-form-driven enforcement. Using these tags, we then select several communities for deeper case studies. We also use these tags to make sense of learning-based subreddits from the popular community site Reddit, which offers an API for programmatic analysis. Based on these techniques, we offer findings about the performance of informal learning communities at scale and issue a call to include these envi-ronments more fully in future research on learning at scale.

Existing attempts to foster a greater sense of community in online education have largely focused on direct interactions among students in peer review, forums, and other mechanisms. In this paper, we pose a new design challenge for learning at scale: peripheral community. Peripheral community is the sense of community derived from peripheral interactions in which a student has visibility into others' behaviors without a direct, intentional interaction occurring between the students. We argue for the value of peripheral community by examining opportunities for such visibility in residential learning environments. We then explore possible ways to supply peripheral community, both in the form of new initiatives and in reinterpretations of existing interventions as fostering peripheral community.

This workshop proposes specifically soliciting contributions and presentations from initiatives, programs, and platforms around the world. While many of these may already be presented at the full conference, we are also interested in more casual experience reports, case studies, and background presentations from individuals more closely acquainted with how learning at scale initiatives-including MOOCs, for-credit degree programs, informal learning environments, government initiatives, and so on-have unique needs and opportunities based on their local context. We refer to this as Global Learning @ Scale. For the purposes of this workshop, we take two views of Global Learning @ Scale.

As online education proliferates, one concern that has been raised is that it may fail to capture desirable emergent phe-nomena from on-campus programs. Student community is one example of such a phenomenon: on-campus student communities thrive based on synchronous collocation. An online program might be designed to capture all deliberate constructs in an on-campus program, but there may be beneficial side effects of synchronous collocation that are not apparent. In this work, we examine the issue of social isolation in an online graduate program. By happenstance, three studies were conducted in relative isolation looking at social isolation from different angles. The first study exam-ined trajectories in social presence as a semester proceeded. The second study developed an understanding of students' needs with regard to community in an online program. The third study tested out an immersive virtual environment to try to improve students' sense of connectedness. Combin-ing their findings, we find compelling evidence of the exist-ence of a Synchronicity Paradox in online education: stu-dents desire synchronicity to form strong social communi-ties, and yet part of the chief appeal of these online pro-grams is their asynchronicity. In light of this finding, we provide design guidelines for how synchronicity may be reintroduced into asynchronous programs without sacrific-ing the benefits of asynchronicity. More specifically, we propose that scale itself may be the key to building emer-gent synchronicity.

This paper presents two approaches using Simulated Annealing and a genetic algorithm to create optimal curricula. The method generates a customized course selection and schedule for individual students enrolled in a large online graduate program in computer science offered by a major public research institution in the United States.

Following the initial proliferation of Massive Open Online Courses (MOOCs), a more recent trend has emerged toward offering "Affordable Degrees at Scale" or "Large, Internet-Mediated Asynchronous Degrees". In this research, we set out to understand this space: the range in tuition costs for these programs, the variety of admissions standards, and the types of assessments used to evaluate these non-traditional students. In the process, however, we found that in many ways, these programs may not be as new as we initially perceived: similarly-priced online programs have existed from traditional universities for years. In this research, we explore these two questions: what are these new degrees at scale, and how do they actually differ from traditional programs? To explore this, we collected materials for 35 MOOC-based graduate degrees and numerous non-MOOC-based comparable degrees. We then explored the patterns in tuition, admissions requirements, and syllabus information. In this paper, we report the trends we identified in MOOC-based degrees, and attempt to answer the question: what makes these programs different from non-MOOC-based online programs of the past? Ultimately, we find that this new era of programs is similar in many observable ways.

Large-scale educational settings have been common domains for affect detection and recognition research. Most research emphasizes improvements in the accuracy of affect measurement to enhance instructors' efficiency in managing large numbers of students. However, these technologies are not designed from students' perspectives, nor designed for students' own usage. To identify the unique design considerations for affect sensors that consider student capacities and challenges, and explore the potential of affect sensors to support students' self-learning, we conducted semi-structured interviews and surveys with both online students and on-campus students enrolled in large in-person classes. Drawing on these studies we: (a) propose using affect data to support students' self-regulated learning behaviors through a "scaling for empowerment'' design perspective, (b) identify design guidelines to mitigate students' concerns regarding the use of affect data at scale, (c) provide design recommendations for the physical design of affect sensors for large educational settings.

Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program's accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program's impact on the gender divide in computing. We also examine older students' tendency towards intrinsic motivation to pursue an MSCS degree.

Peer advising in education, which involves students providing fellow students with course advice, can be important in online student communities and can provide insights into potential course improvements. We examine reviews from a course review web site for online graduate programs. We develop a coding scheme to analyze the free text portion of the reviews and integrate those findings with students' quantitative ratings of each course's overall score, difficulty, and workload. While reviews focus on subjective evaluation of courses, students also provide feedback for instructors, personal context, advice for other students, and objective course descriptions. Additionally, the average review varies by course overall score, difficulty, and workload. Our research examines the importance of student communities in online education and peer advising at scale.

In 2014, Georgia Tech launched the first for-credit MOOC-based graduate degree program. In the five years since, the program has proven generally successful, enrolling over 14,000 unique students, and several other similar programs have followed in its footsteps. Existing research on the program has focused largely on details of individual classes; program-level research, however, has been scarce. In this paper, we delve into the program-level details of an at-scale Master's degree, from the story of its creation through the data generated by the program, including the numbers of applications, admissions, matriculations, and graduations; enrollment details including demographic information and retention patterns; trends in student grades and experience as compared to the on-campus student body; and alumni perceptions. Among our findings, we note that the program has stabilized at a retention rate of around 70%; that the program's growth has not slowed; that the program has not cannibalized its on-campus counterpart; and that the program has seen an upward trend in the number of women enrolled as well as a persistently higher number of underrepresented minorities than the on-campus program. Throughout this analysis, we abstract out distinct lessons that should inform the development and growth of similar programs.

Online classes and degree programs continue to grow in popularity, in part due to the increased convenience and accessibility of education that technology has provided in recent years. As online education scales upwards and outwards, there is an increased need to provide students with an engaging and collaborative learning experience. In some online learning environments, student collaboration is perceived to be more difficult than it is in a physical classroom setting due to cultural or geographic distance between students. In particular, online class lectures often lack the collaborative spirit seen in most in-person classroom lectures. To improve upon the online classroom experience, this project first examines the benefits and drawbacks of several in-person and online lecture delivery techniques, then proposes an online lecture platform that allows students to facilitate their own collaborative classrooms on-demand through a semi-synchronous viewing area and chatroom.

Large undergraduate CS courses receive thousands of code submissions per term. To help make sense of the large quantities of submissions, projects have emerged to dynamically cluster student submissions by approach for writing scalable feedback, tailoring hints, and conducting research. However, relatively little attention has been paid to the value of these tools for informing revision to core course materials and delivery methods. In this work, we applied one such technology-Sense, the eponymous product of its company-to an online CS1 class delivered simultaneously for credit to on-campus students and for free to MOOC students. Using Sense, we clustered student submissions to around 70 problems used throughout the course. In this work, we discuss the value of such clustering, the surprising trends we discovered through this process, and the changes made or planned to the course based on the results. We also discuss broader ideas on injecting clustering results into course design.

We outline how we detected programming plagiarism in an introductory online course for a master's of science in computer science program, how we achieved a statistically significant reduction in programming plagiarism by combining a clear explanation of university and class policy on academic honesty reinforced with a short but formal assessment, and how we evaluated plagiarism rates before and after implementing our policy and assessment.

As online offerings have matured and expanded, new efforts have recently been devoted to opening fully-accredited online versions of traditional on-campus classes. Such classes may be offered to on-campus students for greater flexibility during busy semesters, or to allow them to continue to make progress toward their degrees during internships or semesters away from campus. This trend intersects with a growing CS for All movement that sees more and more non-computer science majors enrolling in CS classes. As online offerings expand, it is important for us to understand who enrolls in online sections and the reasons for their choices, both to make sure that learning outcomes are similar across different delivery mechanisms, and to take advantage of opportunities to tailor course content to specific audiences. In this analysis, we look at two versions, one online and one traditional, of a CS1 class offered at a major public research university. We investigate demographic, motivational, and experiential components to identify which types of students are most likely to select each version and what implications this decision has for student success and course design.

In recent years, non-credit options for learning at scale have outpaced for-credit options. To scale for-credit options, workflows and policies must be devised to preserve the characteristics of accredited higher education---such as the presumption of human evaluation and an assertion of academic integrity---despite increased scale. These efforts must follow as well with shifting from offering isolated courses (or informal collections thereof) to offering full degree programs with additional administrative elements. We see this shift as one from Massive Open Online Courses (MOOCs) to Large, Internet-Mediated Asynchronous Degrees (Limeades). In this work, we perform a qualitative research study on one such program that has scaled to 6,500 students while retaining full accreditation. We report a typology of policies and workflows employed by the individual classes to deliver this experience.

If a MOOC is to qualify for equal credit as an existing on-campus offering, students must achieve comparable outcomes, both educational and attitudinal. We have built a MOOC for teaching CS1 with the intent of offering it for degree credit. To test its eligibility for credit, we delivered it as an online for-credit course for two semesters to 197 on-campus students who selected the online version rather than a traditional version. We compared the demographics, outcomes, and experiences of these students to the 715 students in the traditional version. We found the online students more likely to be older; to be underrepresented minorities; and to have previously failed a CS class. We then found that our online students attained comparable learning outcomes to students in the traditional section. Finally, we found that our online students perceived the online course quality more positively and required less time to achieve those comparable learning outcomes.

Traditionally, education relies on a linear relationship between enrollment and staff; rising enrollment dictates increases to staff with some expertise (such as teaching assistants, TAs) for evaluation. This relationship is expensive, so learning at scale has largely deemphasized expert evaluation and feedback. Two organizations, though, have used different models to scale up class size online while retaining this expert evaluation and feedback. In this paper, we analyze the methods these two organizations have used to increase enrollment while preserving scalability and feedback. We observe an academic program has scaled feedback with traditional TAs by relying on unique characteristics of its student body, while a commercial program has done so with a novel, network-based model. These successes show the potential of learning from experts at scale.

In January 2017, Georgia Tech launched a new online section of its CS1301: Introduction to Computing class. The course, offered both as a for-credit course to on-ground students and as an open MOOC, built on four unique design principles: congruency, adaptivity, modularity, and personalization. In this short paper, we describe the background of the course, the definitions of these design principles, and their application to the course design.

In Fall 2014 we offered an online course on Knowledge-Based Artificial Intelligence (KBAI) to about 200 students as part of the Georgia Tech Online MS in CS program. By now we have offered the course to more than 1000 students. We describe the design, development and delivery of the online KBAI class in Fall 2014.

In October 2014, one-time MOOC developer Udacity completed its transition from primarily producing massive, open online courses to producing job-focused, project-based microcredentials called "Nanodegree" programs. With this transition came a challenge: whereas MOOCs focus on automated assessment and peer-to-peer grading, project-based microcredentials would only be feasible with expert evaluation. With dreams of enrolling tens of thousands of students at a time, the major obstacle became project evaluation. To address this, Udacity developed a system for hiring external experts as project reviewers. A year later, this system has supported project evaluation on a massive scale: 61,000 projects have been evaluated in 12 months, with 50% evaluated within 2.5 hours (and 88% within 24 hours) of submission. More importantly, students rate the feedback they receive very highly at 4.8/5.0. In this paper, we discuss the structure of the project review system, including the nature of the projects, the structure of the feedback, and the data described above.

Large classes, both online and residential, typically demand many graders for evaluating students' written work. Some classes attempt to use autograding or peer grading, but these both present challenges to assigning grades at for-credit institutions, such as the difficulty of autograding to evaluate free-response answers and the lack of expert oversight in peer grading. In a large, online class at Georgia Tech in Summer 2015, we experimented with a new approach to grading: framing graders as meta-reviewers, charged with evaluating the original work in the context of peer reviews. To evaluate this approach, we conducted a pair of controlled experiments and a handful of qualitative analyses. We found that having access to peer reviews improves the perceived quality of feedback provided by graders without decreasing the graders' efficiency and with only a small influence on the grades assigned.

Many ongoing efforts in online education aim to increase accessibility through affordability and flexibility, but some critics have noted that pedagogy often suffers during these efforts. In contrast, in the low-cost for-credit Georgia Tech Online Masters of Science in Computer Science (OMSCS) program, we have observed that the features that make the program accessible also lead to pedagogical benefits. In this paper, we discuss the pedagogical benefits, and draw a causal link between those benefits and the factors that increase the program's accessibility.

Despite the ubiquitous use of videos in online learning and enormous literature on designing online learning, there has been relatively little research on what pedagogical strategies should be used to make the most of video lessons and what constitutes an effective video for student learning. We experimented with a model of incorporating four pedagogical strategies, four instructional phases, and four production guidelines-in designing and developing video lessons for an online graduate course. In this paper, we share our experience as well as students' perceptions of their effectiveness. We also discuss what needs to be done for future research.

Cheating in computer science classes can damage the reputation of institutions and their students. It is therefore essential to routinely authenticate student submissions with available software plagiarism detection algorithms such as Measure of Software Similarity (MOSS). Scaling this task for large classes where assignments are repeated each semester adds complexity and increases the instructor workload. The MOSS Tool for Addressing Plagiarism at Scale (MOSS-TAPS), organizes the MOSS submission task in courses that repeat coding assignments. In a recent use-case in the Online Master of Science in Computer Science (OMSCS) program at the Georgia Institute of Technology, the instructor time spent was reduced from 50 hours to only 10 minutes using the managed submission tool design presented here. MOSS-TAPS provides persistent configuration, supports a mixture of software languages and file organizations, and is implemented in pure Java for cross-platform compatibility.

This paper presents the design and evaluation of a set of intelligent tutoring agents constructed to teach teams of students an authentic process of inquiry-driven modeling. The paper first presents the theoretical grounding for inquiry-driven modeling as both a teaching strategy and a learning goal, and then presents the need for guided instruction to improve learning of this skill. However, guided instruction is difficulty to provide in a one-to-many classroom environment, and thus, this paper makes the case that interaction with a metacognitive tutoring system can help students acquire the skill. The paper then describes the design of an exploratory learning environment, the Modeling and Inquiry Learning Application (MILA), and an accompanying set of metacognitive tutors (MILA--T). These tools were used in a controlled experiment with 84 teams (237 total students) in which some teams received and interacted with the tutoring system while other teams did not. The effect of this experiment on teams' demonstration of inquiry-driven modeling are presented.

Scientists use both conceptual models and executable simulations to help them make sense of the world. Models and simulations each have unique affordances and limitations, and it is useful to leverage their affordances to mitigate their respective limitations. One way to do this is by generating the simulations based on the conceptual models, preserving the capacity for rapid revision and knowledge sharing allowed by the conceptual models while extending them to provide the repeated testing and feedback of the simulations. In this paper, we present an interactive system called MILAfiS for generating agent-based simulations from conceptual models of ecological systems. Designed with STEM education in mind, this user-centered interface design allows the user to construct a Component-Mechanism-Phenomenon conceptual model of a complex system, and then compile the conceptual model into an executable NetLogo simulation. In this paper, we present the results of a pilot study with this interface with about 50 middle school students in the context of learning about ecosystems.

In this paper, we describe the evolution of an interactive technology called the Ecological Modeling Toolkit (EMT) that supports learning about complex ecological systems in middle school science. Authentic learning of science is facilitated by imitation, rehearsal and understanding of real-world scientific practices such as observation, experimentation, problem formulation, hypothesis testing, and model construction and revision. We illustrate how the tools in EMT work together to support many real-world scientific practices such as model construction, simulation and revision, and scaffold others such as observation, problem formulation and hypothesis testing.

This paper presents Tangible Tracking Table, an interactive tabletop display, and Optical Chess, a strategy game. We discuss the design and implementation of both systems and report our evaluation game play sessions with young adults, with a special focus on how the Tangible Tracking Table enhances interaction over a point-and-click interface.