Abstract The importance of programming education has led to dedicated educational programming environments, where users visually arrange block-based programming constructs that typically control graphical, interactive game-like programs. TheScratchprogramming environment is particularly popular, with more than 90 million registered users at the time of this writing. While the block-based nature ofScratchhelps learners by preventing syntactical mistakes, there nevertheless remains a need to provide feedback and support in order to implement desired functionality. To support individual learning and classroom settings, this feedback and support should ideally be provided in an automated fashion, which requires tests to enable dynamic program analysis. In prior work we introducedWhisker, a framework that enables automated testing ofScratchprograms. However, creating these automated tests forScratchprograms is challenging. In this paper, we therefore investigate how to automatically generateWhiskertests. Generating tests forScratchraises important challenges: First, game-like programs are typically randomised, leading to flaky tests. Second,Scratchprograms usually consist of animations and interactions with long delays, inhibiting the application of classical test generation approaches. Thus, the new application domain raises the question of which test generation technique is best suited to produce high coverage tests capable of detecting faulty behaviour. We investigate these questions using an extension of theWhiskertest framework for automated test generation. Evaluation on common programming exercises, a random sample of 1000Scratchuser programs, and the 1000 most popularScratchprograms demonstrates that our approach enablesWhiskerto reliably accelerate test executions, and even though manyScratchprograms are small and easy to cover, there are many unique challenges for which advanced search-based test generation using many-objective algorithms is needed in order to achieve high coverage.

Search-based test generation commonly uses fitness functions based on branch distances, i.e., estimations of how close conditional statements in a program are to evaluating to true or to false. When conditional statements depend on Boolean variables or Boolean-valued methods, the branch distance metric is unable to provide any guidance to the search, causing challenging plateaus in the fitness landscape. A commonly proposed solution is to apply testability transformations, which transform the program in a way that avoids conditional statements from depending on Boolean values. In this paper we introduce the concept of Certainty Booleans, which encode how certain a true or false Boolean value is. Using these Certainty Booleans, a basic testability transformation allows to restore gradients in the fitness landscape for Boolean branches, even when Boolean values are the result of complex interprocedural calculations. Evaluation on a set of complex Java classes and the EvoSuite test generator shows that this testability transformation substantially alters the fitness landscape for Boolean branches, and the altered fitness landscape leads to performance improvements. However, Boolean branches turn out to be much rarer than anticipated, such that the overall effects on code coverage are minimal.

Block-based programming languages like SCRATCH have become increasingly popular as introductory languages for novices. These languages are intended to be used with a “tinkering” approach which allows learners and teachers to quickly assemble working programs and games, but this often leads to low code quality. Such code can be hard to comprehend, changing it is error-prone, and learners may struggle and lose interest. The general solution to improve code quality is to refactor the code. However, SCRATCH lacks many of the common abstraction mechanisms used when refactoring programs written in higher programming languages. In order to improve SCRATCH code, we therefore propose a set of atomic code transformations to optimise readability by (1) rewriting control structures and (2) simplifying scripts using the inherently concurrent nature of SCRATCH programs. By automating these transformations it is possible to explore the space of possible variations of SCRATCH programs. In this paper, we describe a multi-objective search-based approach that determines sequences of code transformations which improve the readability of a given SCRATCH program and therefore form refactorings. Evaluation on a random sample of 1000 SCRATCH programs demonstrates that the generated refactorings reduce complexity and entropy in 70.4% of the cases, and 354 projects are improved in at least one metric without making any other metric worse. The refactored programs can help both novices and their teachers to improve their code.