This Research Full Paper investigates the use of software testing in introductory programming courses (CS1 and/or CS2) as a supportive and facilitating technique for novice students, identifying adopted strategies and tools used. The literature commonly emphasizes problem-solving skills in programming education or strategies for intervening in educational practices. This paper conducts a Systematic Literature Review (SLR) and aims to understand the impact of introducing tests on novice students' comprehension of programming problems. More research is needed to understand how to improve programming teaching materials and methodologies and how novice students absorb all of this. A study was conducted to answer the following research questions: (RQ1) How are tests being used to facilitate novice students' understanding of programming problems? Additionally, (RQ2) What strategies and tools are being adopted to incorporate software testing into teaching programming for beginner students? The responses highlighted the challenges faced in teaching CS1 and/or CS2, evidencing controlled experiments, alternative methods, and tools used. Finally, we critically analyzed the impact on this target audience and found through the research that there is no consensus on the software testing skills necessary to learn programming. Therefore, these skills were grouped as follows: (1) skills related to introducing software testing tools - consolidated or new - and (2) skills related to education with testing activities. By cross-referencing the information, improvements in student performance can be seen when software tests are integrated into the learning process. Introducing tests by customizing materials adopted in programming courses, activities, and evaluations can promote more effective learning for this group.

This Research-to-Practice Full Paper examines the Bebras Challenge tasks, a Computational Thinking (CT) contest that does not require programming knowledge. Numerous studies in the last decade shed light on disseminating CT practices. However, there is still a gap in understanding CT as a construct, that is, CT as a complex cognitive dimension. Furthermore, few studies analyze how many constructs can be recognized inside CT by empirical data. Moreover, algorithmic thinking is one of the essential skills in CT, even when we do not apply programming activities. In this exploratory paper, we investigate the Bebras Challenge tasks. Our primary aim is to figure out how many and which are the constructs associated with CT skills derived from data, especially related to algorithmic thinking. The results point out three primary constructs supported by data. We claim that these constructs are a mix of CT skills, mainly algorithmic thinking, abstraction, generalization, evaluation, and pattern recognition. In addition, our study provides additional evidence supporting the idea that algorithmic thinking can be explored by different perspectives beyond “a set of order instructions”, even without programming practices. Thus, this result reinforces the notion that CT can be understood as a mix of several skills to solve problems. Moreover, algorithmic thinking is a complex cognitive process of thought which may be explored through different strategies. We believe that these findings may help recognize which constructs and abilities underline CT, primarily algorithmic thinking without programming activities.

This full paper, categorized as Research-to-Practice, presents an investigation on the role of source code vocabulary in introductory programming learning practices. According to software engineering literature, source code vocabulary is a fundamental aspect of software quality as it contributes to software readability and eases maintenance and evolution tasks. Furthermore, some studies show that identifier naming and comments in the code tend to use terms that reflect the software problem domain specification and can be used, for instance, to localize feature. In this paper, we build upon this knowledge of software engineering and investigate how we can take advantage of it in computer science education, specially on the teaching of programming. We conducted a twofold empirical study in an introductory programming course to investigate: (1) How to give automated feedback about identifier naming, aiming to improve software readability, and (2) To what extent source code vocabulary reflects problem specification comprehension, which is crucial to effective programming problem-solving. In the first study, we found that 51.7% of the students improved their source code vocabulary, after receiving such feedback. In the second study, the results showed that students tend to manage to better comprehend the programming problem being solved when their code identifier names are connected to the related description. It is a promising indicative that we can use this information to assess the problem requirements comprehension, which is a fundamental step in the programming problem-solving cycle. The main contribution of this paper is to shed light on the source code vocabulary and promote its role in the programming teaching and learning scenario. We present some evidence that students' source code vocabulary is a rich information source about their understanding and we can use it to produce formative feedback.

While several approaches have been proposed to assess computational thinking (CT) abilities, it is still unclear how many and which these abilities are. Despite the undisputed importance of assessment, the fact is that there is not enough evidence on which abilities are merely theoretical and which can be empirically observed in the context of CT. This paper is part of a larger investigation in which we try to answer a simple question: can CT abilities be quantitatively defined and measured? In this particular study, we try to answer a simple question: How many CT abilities can be empirically observed using factor analysis? We approach this question, using a dataset containing answers of 1564 students from Lithuania in the Bebras Challenge from 2015. Firstly, we used confirmatory factor analysis to verify a theory that claims that five CT abilities are assessed by the contest. Our analysis shows the theory is not statistically supported. Secondly, we used principal component analysis as an exploratory analysis to try to derive an appropriate number of factors from the data. Surprisingly, the analysis suggests there are only two main recognizable factors. Finally, we briefly discuss these factors and hypothesize that the first is called evaluation ability, which would include abstraction, generalization, and decomposition, while factor 2 is algorithmic thinking and logical reasoning.

In recent years, the use of game elements in non-game contexts has been widely adopted in different contexts, one of which education. Concerned with investigating the effect of gamification on student engagement in the context of an undergraduate Introductory Programming course that applied that Mastery Learning approach, a gamified web application was developed and integrated to a pre-existing online platform used to conduct the syllabus. N = 60 students were randomly assigned to one of two groups: an experimental group with access to the gamified platform and a control group without access to it. The randomization guaranteed a similar number of students in relation to advancement in the course, in each group. The main features explored in the gamified approach were anonymous weekly leaderboards, many categories of badges and personal record-tracking. Each student profile was private. A first 90-minute experiment showed that the experimental group solved 37% more exercises than the control group, on average. From that point onward, student activity was monitored for a period of four weeks, until the end of the course. Though student-activity decreased over time as students completed the course, the experimental group increasingly solved more exercises per week compared to the control group (at least 46% more, on average). After the course was concluded, the experimental group answered a survey which showed that the badges were the feature that motivated them the most. This research's source code is available and may be easily replicated or incremented.

This Research Full Paper presents results from gamified in-class interventions with leaderboards. Generally, it is assumed that student expertise is directly related to deliberate and repeated practice. However, finding techniques that foster student engagement in problem-solving may not be the easiest of tasks. To that end, gamification, also known as the use of game design elements in non-game contexts, has proved to be effective in educational scenarios. Concerned with stimulating student engagement in Computer Science, this research aimed to observe the effect of the insertion of leaderboards in short-length in-class experiments performed in two different terms of undergraduate IP courses. N = 84 students were randomly allocated into one of two types of group: experimental groups with leaderboards; and control groups without knowledge of nor access to leaderboards. Both experiments showed that students in the experimental groups tried and completed more programming problems, on average, than those from their respective control groups. P-values below α = 5% indicated a real impact from the leaderboard in students' performances. This research may benefit educators responsible for IP courses that have an online platform at their disposal, by introducing a design of leaderboard that incites student engagement, at least for short-length activities.

Several instruments have been used for assessing Computational Thinking (CT) abilities. In this exploratory and preliminary study, we investigate how appropriate the Bebras challenge is as an instrument to assess and measure CT abilities. Bebras is an international challenge whose goal is to promote Computer Science and CT. The test can be answered without any prior knowledge on computer science. Our broad research question is whether we can evolve Bebras into a full fledged instrument to assess and measure CT abilities. In this paper, we instantiate a few more specific research questions: Is Bebras performance a good predictor of success for students within programming courses? Is there any correlation between Bebras performance and students' grades? Do students improve their performance in Bebras tests when exposed to the contents of a programming course? Our dataset consists of the grades of 138 students who attended introductory programming courses at two Brazilian universities and their performance in two simulated Bebras tests. The first test was applied at the beginning of the term and therefore before any exposition to programming classes. The second one was applied at the end of the term. The results suggest that the performance on Bebras is only moderately correlated to the student grades. We conclude that it is not very likely that CT measures can be derived from the Bebras test as it is currently designed. While further research is needed on how we can leverage the Bebras effort to extend it into a CT assessment instrument, we performed a preliminary study on the use of Item Response Theory (IRT) as a means to improve the selection of questions and the design of the test. We expect the results of this research can contribute both to the development and discussion on CT assessment as well as to the Bebras effort to educate CT.

In this paper, we analyzed whether the ability of students to solve problems can be influenced by the implementation of Computational Thinking (CT) in the teaching of mathematics. In this sense, we performed a quasi-experiment with students in the Brazilian's Basic Education Regular System (particularly in a group of students around fifteen years old) comparing their performance after being trained with practical activities using exercises in more or less conformity with well known CT concepts and capabilities. Considering the quantitative results obtained, the experimental group outperformed the control group, which worked with exercises in more and less conformity with CT, respectively. The results were statistically significant, suggesting that bringing together CT and mathematics through proper adjustments of classroom practices can have a positive influence on students abilities of solving problems. In order to strengthen this conclusion we also performed a qualitative analysis comprising students and the teacher involved in the quasi-experiment that reinforced our quantitative results.

The interest on effective strategies to improve the learning process increased over the years. The usual practice is to know the characteristics of a student and how those can affect the learning style. The style describes how learners collect, select, interpret, organize and store information. This paper overviews the application of learning styles and how those affect the teaching of introductory programming. We performed a systematic mapping study to improve our knowledge about student characteristics in the teaching of programming. Twenty-one papers were relevant for our research. The main findings are: i) the Soloman-Felder Index is the most used model of learning style; ii) studies focus on the effects of learning styles on student performance; iii) instructors apply active approaches to motivate a different kind of learners. We conclude that learning style does affect a student's ability to learn. For this reason, teachers should incorporate those styles into their pedagogical material to cater to the majority of students needs.

Several initiatives have been created to promote Computational Thinking (CT) abilities in students. There are multiple approaches of assessing CT and wide abilities and skills involved. However, the evidence on how to assess CT has not yet been systematically grouped or reviewed. The goal of our study is to identify and classify approaches to promote CT and the different ways of assessing CT abilities. To achieve this goal, a systematic mapping study was planned and executed. The results reveal that: (i) programming courses are the most common pedagogical approaches to promote CT for K-12 students; (ii) multiple skills are involved in CT, but solving problems, algorithms, and abstraction are most common abilities assessed; and (iii) codes and multi-choice questionnaires are the most common artifacts for assessing CT abilities. This study points out to the fact that there are open questions for exploring and developing new researches for promoting and assessing CT abilities.

Most introductory programming courses count on automated assessment systems (AAS) to support practical programming assignments and give fast feedback. AAS usually rely on tests results to check the program's functional correctness to provide feedback to students. Novice programmers, however, may find it difficult to map such feedback to the root failures' cause in their programs. It can be even more frustrating when the code is “almost right”. In this paper we investigated the use of a fault localization technique on programs produced by students of introductory programming. Our proposed approach is grounded on spectrum-based fault localization (SBFL). The results of our empirical study showed that this lightweight technique is promising. It can be easily adapted to different AAS to generate useful feedback not only to students but also to instructors. We also delineate the scope where SBFL use is jeopardized. The main contribution of this paper is to present the benefits and drawbacks of applying SBFL, in the context of programming learning, as a novel source of information about students' programming assignments faults.

Proper feedback can leverage students to better understand their difficulties and shorten the characteristic program-submit-refactor cycle of programming exercises. The ideal feedback is the result of a human inspection and analysis considering both functional and qualitative aspects of programs produced by students. On the other hand, automated assessment systems can provide rapid, cheap and standardized feedback. In this paper, we focus on measuring aspects of code that instructors usually assess in programming assignments which are deemed unmeasurable: qualitative aspects that go beyond functional correctness. The aim of this work is to produce richer feedback messages that go beyond functional correctness as it involves code quality issues. We found that if an instructor is required to produce a reference solution for a programming assignment, then most of the expectations the instructor has about a student's code quality are concretely present in the reference solution. Based on this idea, we proposed and evaluated a set of candidate quality measures using the assignment's reference solution as a baseline. The results showed that they seem to capture what is usually considered to be the subjective and qualitative aspects of an instructors' assessment. We used these findings to generate feedback and conducted an experiment to evaluate it effectiveness. The results enforce that this kind of feedback stimulates students to improve their quality code in a higher degree than purely functional feedback, yet it still can be fully automated.

Design is often raised in the literature as important to attaining various properties and characteristics in a software system. At least for open-source projects, it can be hard to find evidence of ongoing design work in the technical artifacts produced as part of the development. Although developers usually do not produce specific design documents, they do communicate about design in different ways. In this paper, we provide quantitative evidence that developers address design through discussions in commits, issues, and pull requests. To achieve this, we built a discussions' classifier and automatically labeled 102,122 discussions from 77 projects. Based on this data, we make four observations about the projects: i) on average, 25% of the discussions in a project are about design; ii) on average, 26% of developers contribute to at least one design discussion; iii) only 1% of the developers contribute to more than 15% of the discussions in a project; and iv) these few developers who contribute to a broad range of design discussions are also the top committers in a project.

In introductory programming courses it is common to demand from students exercises based on the production of code. However, it is difficult for the teacher to give fast feedback to the students about the main solutions tried, the main errors and the drawbacks and advantages of certain solutions. If we could use automatic code comparison algorithms to build visualisation tools to support the teacher in analysing how each solution provided is similar or different from another, such information would be able to be rapidly obtained. However, can computers compare students code solutions as well as teachers? In this work we present an experiment in which we have requested teachers to compare different code solutions to the same problem. Then we have evaluated the level of agreement among each teacher comparison strategy and some algorithms generally used for plagiarism detection and automatic grading. We found out a maximum rate of 77% of agreement between one of the teachers and the algorithms, but a minimum agreement of 75%. However, for most of the teachers, the maximum agreement rate was over 90% for at least one of the automatic strategies to compare code. We have also detected that the level of agreement among teachers regarding their personal strategies to compare students solutions was between 62% and 95%, which shows that there may be more agreement between a teacher and an algorithm than between a teacher and one of her colleagues regarding their strategies to compare students' solutions. The results also seem to support that comparison of students' codes has significant potential to be automated to help teachers in their work.

There is a growing interest of the Computer Science education community for including testing concepts on introductory programming courses. Aiming at contributing to this issue, we introduce POPT, a Problem-Oriented Programming and Testing approach for Introductory Programming Courses. POPT main goal is to improve the traditional method of teaching introductory programming that concentrates mainly on implementation and neglects testing. According to POPT, students' skills must be developed by dealing with ill-defined problems, from which students are stimulated to develop test cases in a table-like manner in order to enlighten the problems' requirements and also to improve the quality of generated code. This paper presents POPT and a case study performed in an Introductory Programming course of a Computer Science program at the Federal University of Rio Grande do Norte, Brazil. The study results have shown that, when compared to a Blind Testing approach, POPT stimulates the implementation of programs of better external quality - the first program version submitted by POPT students passed in twice the number of test cases (professor-defined ones) when compared to non-POPT students. Moreover, POPT students submitted fewer program versions and spent more time to submit the first version to the automatic evaluation system, which lead us to think that POPT students are stimulated to think better about the solution they are implementing.

Verifying whether a software meets its functional requirements plays an important role in software development. However, this activity is necessary, but not sufficient to assure software quality. It is also important to check whether the code meets its design specification. Although there exists substantial tool support to assure that a software does what it is supposed to do, verifying whether it conforms to its design remains as an almost completely manual activity. In a previous work, we proposed design tests — test-like programs that automatically check implementations against design rules. Design test is an application of the concept of test to design conformance checking. To support design tests for Java projects, we developed DesignWizard, an API that allows developers to write and execute design tests using the popular JUnit testing framework. In this work, we present a study on the usability and scalability of DesignWizard to support structural conformance checking through design tests. We conducted a qualitative usability evaluation of DesignWizard using the Think Aloud Protocol for APIs. In the experiment, we challenged eleven developers to compose design tests for an open-source software project. We observed that the API meets most developers' expectations and that they had no difficulties to code design rules as design tests. To assess its scalability, we evaluated DesignWizard's use of CPU time and memory consumption. The study indicates that both are linear functions of the size of software under verification.

We present a functional verification methodology that employs hierarchical coloured petri nets (HCPN) to describe the testbench. By this way, we are avoiding the absence of formal techniques concerning the testbench description and keeping a high-level of abstraction that is required in this phase of the project. The hierarchical (de)composition is the solution to deal with large designs. The methodology prescribes a way to (de)compose the testbench that promotes incremental development and reuse of testbench elements. Furthermore, our methodology provides tool support for the testbench creation. Experimental results concerning the functional verification of the MPEG 4 video decoder are presented.

A strategy for exploring distributed state spaces using computational grids that run bag-of-task applications is discussed. The main idea is to use computational grid tools as a layer between the verification tool and the distributed shared resources, aggregating the vast background developed by researchers in grid computing. Hence, the computational grid deals with resource scalability, computational speedup, and reliability in a transparent manner for the verification tool. Experimental results using a state space generation tool for an object-oriented Petri nets and the OurGrid solution show that is possible to achieve speedup applying it to a private network environment. Moreover, when a wide distributed community is considered, the model size can be increased several times.

In this paper we discuss relevant issues of inheritance in the context of object-oriented specification languages with concurrent semantics based on High-Level Petri Nets. A specific formalization, named OO-CPN, is presented to discuss possible approaches to deal with inheritance in such a context.

Groupware systems are inherently concurrent and complex. To formally specify such systems, designers need powerful description tools capable of dealing with concurrency and mechanisms to manage the complexity. In this paper, we present an object-oriented specification language based on Petri nets, and its application to the modeling of a cooperative editor.