A growing number of social science researchers, including educational researchers, have turned to online crowdsourcing platforms such as Prolific and MTurk for their experiments. However, there is a lack of research investigating the quality of data generated by online subjects and how they compare with traditional subject pools of college students in studies that involve cognitively demanding tasks. Using an interactive problem-solving task embedded in an educational simulation, we compare the task engagement and performance based on the interaction log data of college students recruited from Prolific to those from an introductory physics course. Results show that Prolific participants performed on par with participants from the physics class in obtaining the correct solutions. Furthermore, the physics course students who submitted incorrect answers were more likely than Prolific participants to make rushed cursory attempts to solve the problem. These results suggest that with thoughtful study design and advanced learning analytics and data mining techniques, crowdsourcing platforms can be a viable tool for conducting research on teaching and learning in higher education.

This study introduces a new analysis scheme to analyze trace data and visualize students’ self-regulated learning strategies in a mastery-based online learning modules platform. The pedagogical design of the platform resulted in fewer event types and less variability in student trace data. The current analysis scheme overcomes those challenges by conducting three levels of clustering analysis. On the event level, mixture-model fitting is employed to distinguish between abnormally short and normal assessment attempts and study events. On the module level, trace level clustering is performed with three different methods for generating distance metrics between traces, with the best performing output used in the next step. On the sequence level, trace level clustering is performed on top of module-level clusters to reveal students’ change of learning strategy over time. We demonstrated that distance metrics generated based on learning theory produced better clustering results than pure data-driven or hybrid methods. The analysis showed that most students started the semester with productive learning strategies, but a significant fraction shifted to a multitude of less productive strategies in response to increasing content difficulty and stress. The observations could prompt instructors to rethink conventional course structure and implement interventions to improve self-regulation at optimal times.

In an earlier study we showed that small amounts of extra credit offered for early progress on online homework assignments can reduce cramming behavior in introductory physics students. This work expands on the prior study by implementing a planning prompt intervention inspired by Yeomans and Reich's similar treatment. In the prompt we asked students to what degree they intended to earn extra credit offered for early work on the module sequence, and what their plan was to realize their intentions. The survey was assigned for ordinary course credit and due several days before the first extra credit deadline. We found that students who completed the prompt earned on average 0.6 more extra credit points and completed the modules an average of 1.1 days earlier compared to a previous semester. We detect the impact of the survey by creating a multilinear model based on data from students exposed to the intervention as well as students in a previous semester. Data from five homework sequences are included in the model to account for differences between the two semesters that cannot be attributed to the planning prompt intervention.

This paper examines the impact of COVID-19 induced campus closure on university students’ self-regulated learning behavior by analyzing click-stream data collected from student interactions with 70 online learning modules in a university physics course. To do so, we compared the trend of six types of actions related to the three phases of self-regulated learning before and after campus closure and between two semesters. We found that campus closure changed students’ planning and goal setting strategies for completing the assignments, but didn’t have a detectable impact on the outcome or the time of completion, nor did it change students’ self-reflection behavior. The results suggest that most students still manage to complete assignments on time during the pandemic, while the design of online learning modules might have provided the flexibility and support for them to do so.

The Mechanics Reasoning Inventory (MRI) is an assessment instrument specifically designed to assess strategic reasoning skills involving core concepts in introductory Newtonian mechanics. Being an assessment of higher order thinking (as opposed to declarative or rule-based procedural thinking), it is necessary to check whether or not the mental constructs underlying actual student responses correlate with the authors' domain classification, which is the subject of this paper. The instrument consists of three types of problems: whether momentum or energy is conserved in a given situation and why, (partly inspired by the paired what/why questions in Lawson's Classroom Test of Scientific Reasoning), application of Newton's 2nd and 3rd law, and decomposing problems into parts (inspired by Van Domelen's Problem Decomposition Diagnostic). It has been administered 183 times in two MIT courses since 2009. Exploratory Factor Analysis (EFA) revealed that each Lawson pair of questions should be considered as one item, after which it identified four factors among the 21 questions that correspond reasonably well with the intended physics topics, and a fifth factor correlated with the concept of circular motion, a difficult topic for students (even though not viewed as a core principle by the designers). We discuss why 6 of the items classified under factors that differed from the expert assignments. There was no strong indication that the students answered each of different problem types similarly, which is a hallmark of students using novice heuristics rather than reasoning based on physical principles to answer the questions.

The study presented in this paper deals with copying answers in MOOCs. Our findings show that a significant fraction of the certificate earners in the course that we studied have used what we call harvesting accounts to find correct answers that they later submitted in their main account, the account for which they earned a certificate. In total, around 2.5% of the users who earned a certificate in the course obtained the majority of their points by using this method, and around 10% of them used it to some extent. This paper has two main goals. The first is to define the phenomenon and demonstrate its severity. The second is characterizing key factors within the course that affect it, and suggesting possible remedies that are likely to decrease the amount of cheating. The immediate implication of this study is to MOOCs. However, we believe that the results generalize beyond MOOCs, since this strategy can be used in any learning environments that do not identify all registrants.

We report the one of the first applications of treatment/control group learning experiments in MOOCs. We have compared the efficacy of deliberate practice-practicing a key procedure repetitively-with traditional practice on "whole problems". Evaluating the learning using traditional whole problems we find that traditional practice outperforms drag and drop, which in turn outperforms multiple choice. In addition, we measured the amount of learning that occurs during a pretest administered in a MOOC environment that transfers to the same question if placed on the posttest. We place a limit on the amount of such transfer, which suggests that this type of learning effect is very weak compared to the learning observed throughout the entire course.