Autograders are programs written to analyze student work from formative assessments and produce both grades and constructive feedback for the benefit of instructors and/or students. Many strategies can be used to develop these programs. This paper demonstrates a strategy based on reflection that allows test cases to examine the internal state of objects and to create objects with arbitrary internal states. We report on our experiences using this technique in a CS2 course, where we found that students given autograders based on this strategy produce more correct solutions than those given autograders that rely on the public interfaces of student-written classes.

Autograders have become an invaluable tool for instructors of computer programming courses. They not only ease the burden of manually grading many assignments, but more importantly provide students with a way to receive immediate feedback about their work giving them opportunities to revise and resubmit. However, autograder messages suffer from the same problem as compiler messages; they are often confusing and unhelpful to the students they are trying to help. This is particularly true for novice programmers.

We describe HTN‐MAKER, an algorithm for learning hierarchical planning knowledge in the form of task‐reduction methods for hierarchical task networks (HTNs). HTN‐MAKER takes as input a set of planning states from a classical planning domain and plans that are applicable to those states, as well as a set of semantically annotated tasks to be accomplished. The algorithm analyzes this semantic information to determine which portion of the input plans accomplishes a particular task and constructs task‐reduction methods based on those analyses. We present theoretical results showing that HTN‐MAKER is sound and complete. Our experiments in five well‐known planning domains confirm the theoretical results and demonstrate convergence toward a set of HTN methods that can be used to solve any problem expressible as a classical planning problem in that domain, relative to a set of goal types for which tasks have been defined. In three of the five domains, HTN planning with the learned methods scales much better than a modern classical planner.

We consider how to learn Hierarchical Task Networks (HTNs) for planning problems in which both the quality of solution plans generated by the HTNs and the speed at which those plans are found is important. We describe an integration of HTN Learning with Reinforcement Learning to both learn methods by analyzing semantic annotations on tasks and to produce estimates of the expected values of the learned methods by performing Monte Carlo updates. We performed an experiment in which plan quality was inversely related to plan length. In two planning domains, we evaluated the planning performance of the learned methods in comparison to two state-of-the-art satisficing classical planners, FastForward and SGPlan6, and one optimal planner, HSP*. The results demonstrate that a greedy HTN planner using the learned methods was able to generate higher quality solutions than SGPlan6 in both domains and FastForward in one. Our planner, FastForward, and SGPlan6 ran in similar time, while HSP* was exponentially slower.