This study examines how accurately individual student differences in learning can be predicted from prior student learning activities. Bayesian Knowledge Tracing (BKT) predicts learner performance well and has often been employed to implement cognitive mastery. Standard BKT individualizes parameter estimates for knowledge components, but not for learners. Studies have shown that individualizing parameters for learners improves the quality of BKT fits and can lead to very different (and potentially better) practice recommendations. These studies typically derive best-fitting individualized learner parameters from learner performance in existing data logs, making the methods difficult to deploy in actual tutor use. In this work, we examine how well BKT parameters in a tutor lesson can be individualized based on learners' prior performance in reading instructional text, taking a pretest, and completing an earlier tutor lesson. We find that best-fitting individual difference estimates do not directly transfer well from one tutor lesson to another, but that predictive models incorporating variables extracted from prior reading, pretest and tutor activities perform well, when compared to a standard BKT model and a model with best-fitting individualized parameter estimates.

This paper has been removed because it is redundant with a work the authors presented at an earlier conference whose proceedings ACM had already published. See: http://doi.acm.org/10.1145/1953163.1953254

In this paper, we describe our approach in addressing learning challenges students experience in introductory programming courses. We combine two effective instructional methodologies to help students learn to plan programs prior to writing code: Cognitive Apprenticeship Learning (CAL) and Cognitive Tutors (CT). In the CAL component, the instructor models program planning in class and paper handouts are used to scaffold program planning in homework assignments. In CAL-CT, the program-planning process is also supported by a computer tutor which provides step-by-step feedback and advice. The results show that the combined CAL-CT approach yielded substantial gains over traditional instruction. Its advantage over the CAL-Only approach is also significant.

Gaming the system, attempting to succeed in an interactive learning environment by exploiting properties of the system rather than by learning the material (for example, by systematically guessing or abusing hints), is prevalent across many types of educational software. Past research on why students choose to game has focused on student individual differences. Many student individual differences, including attitudes towards mathematics, have been shown to be associated with gaming, but generally with low correlation. In this paper, we investigate how individual differences between learning environments can increase or decrease the probability of gaming. We enumerate ways intelligent tutor lessons vary from each other, and use data mining to discover hypotheses about how differences in software design and content influence the choice to game the system. We discover a set of tutor features that explain 56% of the variance in gaming, over five times the degree of variance explained in any prior study of student individual differences and gaming. These results provide an important step towards developing prescriptions for designing intelligent tutor software that students game significantly less.

We investigate the prevalence and learning impact of different types of off-task behavior in classrooms where students are using intelligent tutoring software. We find that within the classrooms studied, no other type of off-task behavior is associated nearly so strongly with reduced learning as "gaming the system": behavior aimed at obtaining correct answers and advancing within the tutoring curriculum by systematically taking advantage of regularities in the software's feedback and help. A student's frequency of gaming the system correlates as strongly to post-test score as the student's prior domain knowledge and general academic achievement. Controlling for prior domain knowledge, students who frequently game the system score substantially lower on a post-test than students who never game the system. Analysis of students who choose to game the system suggests that learned helplessness or performance orientation might be better accounts for why students choose this behavior than lack of interest in the material. This analysis will inform the future re-design of tutors to respond appropriately when students game the system.

The advent of second-generation intelligent computer tutors raises an important instructional design question: when should tutorial advice be presented in problem solving? This paper examines four feedback conditions in the ACT Programming Tutor. Three versions offer the student different levels of control over error feedback and correction: (a) immediate feedback and immediate error correction; (b) immediate error flagging and student control of error correction; (c) feedback on demand and student control of error correction. A fourth, No-tutor condition offers no stepby-step problem solving support. The immediate feedback group with greatest tutor control of problem solving yielded the most efficient learning. These students completed the tutor problems fastest, and the three tutor-supported groups performed equivalently on tests. Questionnaires revealed little student preference among the four conditions. These results suggest that students will need explicit guidance to benefit from learning opportunities that arise when they have greater control over tutorial assistance.

We can reliably build “second generation” intelligent computer tutors that are approximately half as effective as human tutors. This paper evaluates two interface enhancements designed to improve the effectiveness of one successful second generation tutor, the ACT Programming Tutor. One enhancement employs animated feedback to make key data structure relationships salient. The second enhancement employs subgoal scaffolding to support students in developing simple programming plans. Both interventions were successful, but had very different impacts on student effort required to achieve mastery in the tutor environment and on subsequent posttest accuracy. These results represent a step forward in closing the gap between computer tutors and human tutors.

An open question in the area of Programming by Demonstration (PBD) is how to best represent the inferred program. Without a way to view, edit, and share programs, PBD systems will never reach their full potential. We designed and implemented two graphical representation languages for a PBD desktop similar to the Apple Macintosh Finder. Although a user study showed that both languages enabled nonprogrammers to generate and comprehend programs, the study also revealed that the language that more closely reflected the desktop domain doubled users' abilities to accurately generate programs. Trends suggest that the same language was easier for users to comprehend. These findings suggest that it is possible for a PBD system to enable nonprogrammers to construct programs and that the form of the representation can impact the PBD system's effectiveness. A paper-and-pencil evaluation of the two versions of the PBD desktop prior to the study supported these finding and provided interesting feedback on the interaction between usability evaluations and user studies. In particular, the comparison of the paper-and-pencil evaluation with the empirical evaluation suggested that nonempirical evaluation techniques can provide guidance into how to interpret empirical data and, in particular, that PBD systems need to provide support for programming-strategy selection in order to be successful.

We present an evaluation of a multimedia educational software system that includes text, graphics, animations, and simulations.When compared with an informationally equivalent control environment that used text and carefully selected still images, we found little evidence that the dynamic presentations enhanced student understanding of the declarative information in this lesson.Furthermore, students cannot be relied on to take fllll advantage of exploratory opportunities in computer-based instruction.These results prescribe further investigation of whether and how computer-based multimedia can be used effectively in education and training.