The flexibility and affordability of online, asynchronous, at-scale degree programs have significantly increased the accessibility of a master's-level graduate education. While studies have been conducted on the general growth of such programs and the quality of the online courses compared to their on-campus counterparts, few (if any) have examined outcomes such as alumni career growth or admission into other graduate programs. This work examines how one large online graduate program in computer science prepared alumni for matriculation into STEM PhD programs. Enrollment data from the National Student Clearinghouse was analyzed to identify key trends in alumni PhD enrollment. Surveys and interviews with program alumni were also conducted to investigate the unique paths that these individuals took to beginning their PhD education. This study finds that the program positively impacted alumni PhD experiences in STEM fields. Alumni noted that involvement with graduate research and coursework were key components in their preparation for a PhD program. These results demonstrate that an affordable, online, asynchronous graduate STEM program can provide non-traditional students with an effective pathway to PhD enrollment. The paper concludes with recommendations for asynchronous, at-scale degree programs seeking to expand their research opportunities for students with a desire to pursue PhD programs.

This study presents an in-depth manual analysis of memory-related bugs within the PyTorch deep learning framework, leveraging a filtered dataset of 1,678 closed issues from the official PyTorch GitHub repository. The selected issues span a three-year period from January 1, 2020, to March 23, 2023, allowing for a comprehensive examination of trends, patterns, and solutions. This study aims to understand the correlations between the characteristics of PyTorch bugs and also the composition of the root causes behind memory bugs. The findings reveal that Correctness and Runtime Error bugs occur most frequently, with a lack of a correlation between Affected Components and Bug Symptoms. Our results highlight the need for more integrated inter-component debugging tools. Furthermore, the findings show that indexing errors occur most frequently among memory bugs. We determine that, to address the severe impact of such memory bugs, there exists a need for more comprehensive and redundant test cases. Through this analysis, this work aims to provide actionable insights for developers to improve the robustness of PyTorch, improving its reliability in machine learning applications.

Foundation language models learn from their finetuning input context in different ways.In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance.These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time.They can be applied either on single language pair translation tasks or massively multilingual translation tasks.Experiments with these techniques demonstrate significant performance improvements up to 3.5 chrF++ on the Flores200 translation benchmark.We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance.Our code is released here.

Odds are we've all used (or tried!) quite a few tools to facilitate efficiency inside and outside of the classroom and empower students to learn more effectively. Some of those tools are perhaps homegrown and unique to one's own institution, but freely available educational technologies abound as well, some in the cloud, some for Macs and PCs, some open-source. And quite a few commercial tools offer free or discounted educational plans as well. In this BoF, we'll begin with a whirlwind tour of the tools we ourselves use, identifying the problems they solve and how well, then quickly open the floor to everyone to share their favorites as well. Along the way, we'll jot down every tool mentioned and share the results. With educational technology an evergreen landscape, this year's list will surely be different from last! Attendees should exit this session with a better understanding of the current landscape, familiarized with innovations they can bring back to their own classes (whether high school, undergraduate, or graduate), without reinventing wheels themselves.

For CS50 at Harvard, we have developed a suite of free, open-source tools to help students with writing, testing, and submitting programming assignments and to help teachers grade those assignments and check them for similarities. help50 parses often-cryptic error messages and explains them in beginner-friendly terms. check50 runs a set of automated tests on students' code, providing feedback on errors. style50 lints students' code, highlighting that don't adhere to the course's style guide. submit50 allows students to submit assignments to a GitHub repository, without students needing to have knowledge of git or version control themselves. And compare50 allows teachers to analyze submissions for similarity, looking for pairs or clusters of submissions that might be the result of improper collaboration. In this workshop, we'll introduce each of these tools and discuss how other teachers can use them in their own classrooms. Along the way, we'll discuss how to use the tools effectively, compare and contrast them with alternatives, identify how the tools have changed students' behavior for the better and for worse, and highlight pedagogical and technological changes we've made to redress the latter.

This workshop offers hands-on experience with a suite of interactive programming environments for teachers and students, each of them cloud-based and free. The first tool is CS50 Sandbox, via which teachers create temporary programming environments quickly and share copies of those sandboxes with students. The second tool is CS50 Lab, via which teachers can create step-by-step programming lessons, providing incremental feedback at each step, enabling students to progress from an empty file (or starter code) to working code, with hints and feedback along the way. And the third tool is CS50 IDE, via which teachers can provide students with their own cloud-based Linux environment. Each of these environments offers students a built-in file browser and code editor, along with a terminal window with shell access to their very own container. The IDE additionally provides an interactive, graphical debugger. Each tool enables students to write programs in any language for which a compiler or interpreter can be installed in the underlying container, including Java and Python. Not only will we demonstrate each tool, we'll discuss lessons learned from having deployed these tools in CS50 at Harvard to hundreds of students on campus and thousands of students online. We'll compare and contrast with commercial and open-source tools. And we'll discuss challenges encountered and best practices adopted.

We present check50, an open-source, extensible tool for assessing the correctness of students' code that provides a simple, functional framework for writing checks as well as an easy-to-use API that abstracts away common tasks, among them compiling and running programs, providing their inputs, and checking their outputs. As a result, check50 has allowed us to provide students with immediate feedback on their progress as they complete an assignment while also facilitating automatic and consistent grading, allowing teaching staff to spend more time giving tailored, qualitative feedback. We have found, though, that since introducing check50 in 2012 in CS50 at Harvard, students have begun to perceive the course's programming assignments as more time-consuming and difficult than in years past. We speculate that the feedback that check50 provides prior to students' submission of each assignment has compelled students to spend more time debugging than they had in the past. At the same time, students' correctness scores are now higher than ever.

Odds are we've all used (or tried!) quite a few tools to facilitate efficiency inside and outside of the classroom and empower students to learn more effectively. Some of these tools are perhaps homegrown and unique to one's own institution, but freely available educational technologies abound as well, some in the cloud, some for Macs and PCs, some open-source. And quite a few commercial tools offer free or discounted educational plans as well. In this BoF, we'll begin with a whirlwind tour of the tools we ourselves use, identifying the problems they solve and how well, then quickly open the floor to everyone to share their favorites as well. Along the way, we'll note every tool mentioned and share the results. Attendees should exit this session with a better understanding of the educational-technology landscape, familiarized with innovations they can bring back to their own classes.

For CS50 at Harvard, we have developed a suite of free, open-source tools to help students with writing, testing, and submitting programming assignments; and to help teachers grade those assignments and check them for plagiarism. help50, a program that parses error messages and provides beginner-friendly advice to interpreting them, helps students understand and resolve often-cryptic compiler errors. check50 runs a set of automated tests on students' code, providing feedback and hints about where students have made errors. style50 lints students' code, highlighting places where it doesn't meet the course's style guide. submit50 allows students to submit assignments to a GitHub repository, without students needing to have knowledge of git or version control themselves. And compare50, an open-source and customizable alternative to Moss, allows teachers to analyze submissions for similarity, looking for pairs or clusters of submissions that might be the result of improper collaboration. The grading and submission tools require only a GitHub account to use, and can serve as free, extensible alternatives to tools like Codio, Gradescope, and Vocareum. In this workshop, we'll introduce each of the tools, and discuss how to use them for your own classroom. To date, each tool has been deployed to hundreds of students on campus and thousands online. Along the way, we'll discuss how to use the tools effectively, compare and contrast them with other options, identify how the tools have changed students' behavior for the better and for worse, and highlight pedagogical and technological changes we've made to redress the latter. Laptop (with Wi-Fi) required. Linux, macOS, or Windows. Latest version of Chrome.

We aspire to teach academic honesty in CS50 at Harvard University not only by addressing academic dishonesty when it occurs but by addressing it before it does. By way of communication, course- and campus-wide awareness of policy, just-in-time prompts, interventional conversations, and problem sets have we tried to preempt submission of plagiarized work. But few interventions have had significant or lasting effects on the number of instances thereof. Most impactful has been the addition of one sentence to the course's syllabus, a "regret clause" that encourages students to come forward within 72 hours of some dishonest act on their part, before the course itself is even aware. While we might zero the work in question in such cases, we commit to not escalating the matter further to the university's honor council, where the outcome might instead be admonishment, probation, or even required withdrawal from the university itself. We instead advise students on how best to move forward and connect them as needed with support structures on campus for academics and mental health. Since 2014 have 89 students invoked the clause, between 1% and 3% of the course's student body each year.

The Nifty Assignments special session is about promoting and sharing the ideas and ready-to-use materials of successful assignments. Each presenter will introduce their assignment, give a quick demo, and describe its niche in the curriculum and its strengths and weaknesses. The presentations (and the descriptions below) merely introduce the assignment. A key part of Nifty Assignments is the mundane but vital role of distributing the materials - handouts, data files, starter code, rubrics, autograders - that make each assignment ready to adopt. Each assignment presented has complete materials freely available on the Nifty Assignments home page nifty.stanford.edu. If you have an assignment that works well and would be of interest to the CSE community, please consider applying to present at Nifty Assignments.