Learning to write high-quality code is a critical skill for novice programmers, but manual code reviews are resource-intensive and do not scale well. Existing automated tools may fail to provide feedback that is relevant to novices, and even many educational tools overlook some novice-specific code quality defects or are difficult to adapt to new settings. To address these limitations, we developed EduLint, a customizable educational linter. It detects a wide range of novice-specific defects -- many of which are not addressed by other tools -- and is easy to adapt to diverse educational settings. To demonstrate this, we describe its deployments in several different courses and seminars, including a large CS1 course where students were required to fix issues EduLint identified. In a quasi-experiment, we observed that students were better at avoiding some of the defects several months after the CS1 course ended. Students also rated EduLint as clear and easy to use. Across all the deployments, EduLint has already delivered code quality feedback on hundreds of thousands of defects to thousands of users.

Code quality is an important aspect of programming education, with duplicate code being a common issue. To help students learn to avoid code duplication, it is useful to provide them with actionable, specific feedback, not just a generic code duplication warning. In this paper, we introduce the concept of diagnosable code duplication, provide an overview of its various types, and propose a framework for automatic detection. We apply the framework to an introductory programming dataset to demonstrate its ability to provide specific feedback and reveal non-trivial differences in detected cases compared to simpler detectors.

Code quality is an important aspect of programming, as quality code is easier to maintain, and code maintenance makes up the majority of software cost. For that reason, code quality should be emphasized in programming education. Previous work has identified many code quality defects commonly made by students. However, the current state lacks a clear organization and prioritization of these defects. In this paper, we propose an organization framework for code quality defects, presenting a catalog that describes 80 defects, with a specific focus on defects frequently encountered in code by novice programmers. To determine which defects are worth pointing out to students, we conducted a survey among 72 educators, who rated the priority with which each defect should be reported to a student. These presented results serve multiple purposes: they facilitate comparison across various research studies, support the advancement of software tools, and offer inspiration for programming education.

Abstract Computer-based learning environments can easily collect student response times. These can be used for multiple purposes, such as modeling student knowledge and affect, domain modeling, and cheating detection. However, to fully leverage them, it is essential to understand the properties of response times and associated caveats. In this study, we delve into the properties of response time distributions, including the influence of aberrant student behavior on response times. We then provide an overview of modeling approaches that use response times and discuss potential applications of response times for guiding the adaptive behavior of learning environments.

Abstract Personalization in online learning environments has been extensively studied at various levels, ranging from adaptive hints during task-solving to recommending whole courses. In this study, we focus on recommending learning activities (sequences of homogeneous tasks). We argue that this is an important yet insufficiently explored area, particularly when considering the requirements of large-scale online learning environments used in practice. To address this gap, we propose a modular rule-based framework for recommendations and thoroughly explain the rationale behind the proposal. We also discuss a specific application of the framework.

To provide practice and assessment of computational thinking, we need specific problems students can solve. There are many such problems, but they are hard to find. Learning environments and assessments often use only specific types of problems and thus do not cover computational thinking in its whole scope. We provide an extensive catalog of well-structured computational thinking problem sets together with a systematic encoding of their features. Based on this encoding, we propose a four-level taxonomy that provides an organization of a wide variety of problems. The catalog, taxonomy, and problem features are useful for content authors, designers of learning environments, and researchers studying computational thinking.

Research on feedback in introductory programming focuses mostly on incomplete and incorrect programs. However, most of the functionally correct programs also contain defects that call for feedback. We analyzed 114,000 solutions to 161 short coding problems in Python and compiled a catalog of 32 defects in code quality. We found that most correct solutions contain some defects and that students do not stop making them if they do not receive targeted feedback. The catalog of defects, together with their prevalence across common topics like expressions, loops, and lists, informs educators which defects to address in which lectures and guides the development of exercises on code quality. Additionally, we describe defect detectors, which can be used to generate valuable feedback to students automatically.

Student models are typically evaluated through predicting the correctness of the next answer. This approach is insufficient in the problem-solving context, especially for student models that use performance data beyond binary correctness. We propose more comprehensive methods for validating student models and illustrate them in the context of introductory programming. We demonstrate the insufficiency of the next answer correctness prediction task, as it is neither able to reveal low validity of student models that use just binary correctness, nor does it show increased validity of models that use other performance data. The key message is that the prevalent usage of the next answer correctness for validating student models and binary correctness as the only input to the models is not always warranted and limits the progress in learning analytics.

Additive Factors Model is a widely used student model, which is primarily used for refining knowledge component models (Q-matrices). We explore the robustness and generalizability of the model. We explicitly formulate simplifying assumptions that the model makes and we discuss methods for visualizing learning curves based on the model. We also report on an application of the model to data from a learning system for introductory programming; these experiments illustrate possibly misleading interpretation of model results due to differences in item difficulty. Overall, our results show that greater care has to be taken in the application of the model and in the interpretation of results obtained with the model.

Ryan Baker presented in a LAK 2019 keynote a list of six grand challenges for learning analytics research. The challenges are specified as problems with clearly defined success criteria. Education is, however, a domain full of ill-defined problems. I argue that learning analytics research should reflect this nature of the education domain and focus on less clearly defined, but practically essential issues. As an illustration, I discuss three important challenges of this type: addressing inherent trade-offs in learning environments, the clarification of methodological issues, and the scalability of system development.

Quantification of the difficulty of problem solving tasks has many applications in the development of adaptive learning systems, e.g., task sequencing, student modeling, and insight for content authors. There are, however, many potential conceptualizations and measures of problem difficulty and the computation of difficulty measures is influenced by biases in data collection. In this work, we explore difficulty measures for introductory programming tasks. The results provide insight into non-trivial behavior of even simple difficulty measures.

Large scale learning systems for introductory programming need to be able to automatically assess the quality of students' performance on programming tasks. This assessment is done using a performance measure, which provides feedback to students and teachers, and an input to the domain, student and tutor models. The choice of a good performance measure is nontrivial, since the performance of students can be measured in many ways, and the design of measure can interact with the adaptive features of a learning system or imperfections in the used domain model. We discuss the important design decisions and illustrate the process of an iterative design and evaluation of a performance measure in a case study.

Block-based environments are today commonly used for introductory programming activities like those that are part of the Hour of Code campaign, which reaches millions of students. These activities typically consist of a static series of problems. Our aim is to make this type of activities more efficient by incorporating adaptive behavior. In this work, we discuss steps towards this goal, specifically a proposal and implementation of a programming game that supports both elementary problems and interesting programming challenges and thus provides an environment for meaningful adaptation. We also discuss methods of adaptivity and the issue of evaluating student performance while solving a problem.

Personalized educational systems adapt their behavior based on student performance. Most student modeling techniques, which are used for guiding the adaptation, utilize only the correctness of student's answers. However, other data about performance are typically available. In this work we focus on response times and wrong answers as these aspects of performance are available in most systems. We analyze data from several types of exercises and domains (mathematics, spelling, grammar). The results suggest that wrong answers are more informative than response times. Based on our results we propose a classification of student performance into several categories.

A personalized learning system needs a large pool of items for learners to solve. When working with a large pool of items, it is useful to measure the similarity of items. We outline a general approach to measuring the similarity of items and discuss specific measures for items used in introductory programming. Evaluation of quality of similarity measures is difficult. To this end, we propose an evaluation approach utilizing three levels of abstraction. We illustrate our approach to measuring similarity and provide evaluation using items from three diverse programming environments.

Personalized educational systems are able to provide learners questions of specified difficulty. Since learners differ, the appropriate level of difficulty may vary and it may be impossible to find an universal setting. We implemented a version of an adaptive educational system for geography practice that allows learners to adjust difficulty of questions. We evaluated this feature using a randomized control experiment. The overall results show only a small effect of the adjustment. A more detailed analysis, however, shows that for some groups of learners the effect can be important, although not necessarily advantageous. The collected data from the experiment provide insight into how to tune question difficulty automatically.

A common personalization approach in educational systems is mastery learning. A key step in this approach is a criterion that determines whether a learner has achieved mastery. We thoroughly analyze several mastery criteria for the basic case of a single well-specified knowledge component. For the analysis we use experiments with both simulated and real data. The results show that the choice of data sources used for mastery decision and setting of thresholds are more important than the choice of a learner modeling technique. We argue that a simple exponential moving average method is a suitable technique for mastery criterion and propose techniques for the choice of a mastery threshold.

Computerized educational systems are increasingly provided as open online services which provide adaptive personalized learning experience. To fully exploit potential of such systems, it is necessary to thoroughly evaluate different design choices. However, both openness and adaptivity make proper evaluation difficult. We provide a detailed report on evaluation of an online system for adaptive practice of geography, and use this case study to highlight methodological issues with evaluation of open online learning systems, particularly attrition bias. To facilitate evaluation of learning, we propose to use randomized reference questions. We illustrate application of survival analysis and learning curves for declarative knowledge. The result provide an interesting insight into the impact of adaptivity on learner behaviour and learning.

Student modeling techniques are evaluated mostly using historical data. Researchers typically do not pay attention to details of the origin of the used data sets. However, the way data are collected can have important impact on evaluation and interpretation of student models. We discuss in detail two ways how data collection in educational systems can influence results: mastery attrition bias and adaptive choice of items. We systematically discuss previous work related to these biases and illustrate the main points using both simulated and real data. We summarize specific consequences for practice -- not just for doing evaluation of student models, but also for data collection and publication of data sets.

We present a "Problem Solving Tutor" - a web-based educational tool for learning through problem solving. The tool contains more than 1,400 problems, mainly introductory programming problems, math and logic puzzles. All problems are interactive and the system gives students immediate feedback on their performance. The tool makes individual predictions of problem solving times and therefore is able to recommend each student a problem of suitable difficulty.

We propose an abstraction-based model checking method which relies on refinement of an under-approximation of the feasible behaviors of the system under analysis. The method preserves errors to safety properties, since all analyzed behaviors are feasible by definition. The method does not require an abstract transition relation to be generated, but instead executes the concrete transitions while storing abstract versions of the concrete states, as specified by a set of abstraction predicates. For each explored transition the method checks, with the help of a theorem prover, whether there is any loss of precision introduced by abstraction. The results of these checks are used to decide termination or to refine the abstraction by generating new abstraction predicates. If the (possibly infinite) concrete system under analysis has a finite bisimulation quotient, then the method is guaranteed to eventually explore an equivalent finite bisimilar structure. We illustrate the application of the approach for checking concurrent programs.

The popularity of object-oriented programming has led to the wide use of container libraries. It is important for the reliability of these containers that they are tested adequately. We describe techniques for automated test input generation of Java container classes. Test inputs are sequences of method calls from the container interface. The techniques rely on state matching to avoid generation of redundant tests. Exhaustive techniques use model checking with explicit or symbolic execution to explore all the possible test sequences up to predefined input sizes. Lossy techniques rely on abstraction mappings to compute and store abstract versions of the concrete states; they explore underapproximations of all the possible test sequences.We have implemented the techniques on top of the Java PathFinder model checker and we evaluate them using four Java container classes. We compare state matching based techniques and random selection for generating test inputs, in terms of testing coverage. We consider basic block coverage and a form of predicate coverage - that measures whether all combinations of a predetermined set of predicates are covered at each basic block. The exhaustive techniques can easily obtain basic block coverage, but cannot obtain good predicate coverage before running out of memory. On the other hand, abstract matching turns out to be a powerful approach for generating test inputs to obtain high predicate coverage. Random selection performed well except on the examples that contained complex input spaces, where the lossy abstraction techniques performed better.

We study the behavior of the random walk method in the context of model checking and its capacity to explore a state space. We describe the methodology we have used for observing the random walk and report on the results obtained. We also describe many possible enhancements of the random walk and study their behavior and limits. Finally, we discuss some practically important but often neglected issues like counterexamples, coverage estimation, and setting of parameters. Similar methodology can be used for studying other state space exploration techniques like bit-state hashing, partial storage methods, or partial order reduction.

We consider the problem of test input generation for code that manipulates complex data structures. Test inputs are sequences of method calls from the data structure interface. We describe test input generation techniques that rely on state matching to avoid generation of redundant tests. Exhaustive techniques use explicit state model checking to explore all the possible test sequences up to predefined input sizes. Lossy techniques rely on abstraction mappings to compute and store abstract versions of the concrete states; they explore under-approximations of all the possible test sequences. We have implemented the techniques on top of the Java PathFinder model checker and we evaluate them using a Java implementation of red-black trees.