Email communication between instructors and students is ubiquitous, and it could be valuable to explore ways of testing out how to make email messages more impactful. This paper explores the design space of using emails to get students to plan and reflect on starting weekly homework earlier. We deployed a series of email reminders using randomized A/B comparisons to test alternative factors in the design of these emails, providing examples of an experimental paradigm and metrics for a broader range of interventions. We also surveyed and interviewed instructors and students to compare their predictions about the effectiveness of the reminders with their actual impact. We present our results on which seemingly obvious predictions about effective emails are not borne out, despite there being evidence for further exploring these interventions, as they can sometimes motivate students to attempt their homework more often. We also present qualitative evidence about student opinions and behaviours after receiving the emails, to guide further interventions. These findings provide insight into how to use randomized A/B comparisons in everyday channels such as emails, to provide empirical evidence to test our beliefs about the effectiveness of alternative design choices.

Procrastination by students may lead to adverse outcomes such as a focus on completion rather than learning or even a failure to complete learning tasks. One common method for motivating students and reducing procrastination is to send reminders with hints and study strategies, but it's not clear if these messages are effective or when is the best time to send them. Randomized A/B comparisons could be used to try different reminders or alternative ideas about how best to get students to start work earlier and, crucially, to measure the impact of these interventions on behaviour. This paper describes an A/B comparison of reminder emails set in a large CS1 course at a research-focused North American university. We found evidence that the email interventions caused a higher proportion of students to attempt the online homework but did not see evidence that these particular emails got students to start early, irrespective of changes to the timing of the reminder. More broadly, these findings illustrate how to use A/B comparisons in educational settings to test ideas about how to help students, and demonstrate the value of using randomized A/B comparisons, even when evaluating actions that seem obviously beneficial, such as reminder emails.