Model-Based Testing (MBT) enables the automated generation of test suites from requirement models. However, the frequent changes in agile development often lead teams to indiscriminately discard existing test cases, undermining the efficiency of Test Case Maintenance (TCM). This practice results in the loss of valuable test artifacts and escalates costs due to redundant test generation. Previous research has explored test case reuse through distance functions, but this strategy often suffers from low precision and misclassification. These issues lead to an excessive number of test cases being incorrectly considered reusable. In this paper, we investigate the use of Large Language Models (LLMs) to improve test case management. Through an empirical study on two industrial systems, we analyzed the performance of 13 well-known LLMs in classifying the impact of use case edits using CoT (Chain of Thought)/ToT (Tree of Thought) prompting and Naive-RAG strategies. Our findings indicate that seven of these models effectively reduced the unnecessary discarding of test cases by accurately identifying high-impact requirement changes, achieving a 7% improvement over distance functions. This resulted in a more precise, reliable, and efficient TCM solution within MBT. However, compared to distance-function-based strategies, LLMs exhibited slightly lower recall, performing 6% worse in test case reuse and reduction information loss.

Model-Based Testing (MBT) can be a valuable tool for software testing, automating test generation from the System Under Test (SUT) models and making the testing process systematic. However, it is common for models to undergo changes during the software lifecycle, which requires adaptive and robust testing strategies. As models evolve, the generated MBT suites require maintenance. While some tests may remain usable, others may become obsolete or require revision. In such scenarios, it is common for parts of an MBT suite to be discarded due to model changes, resulting in additional costs and hindering bug traceability. Therefore, maintaining MBT suites poses a significant challenge to their practical use. This paper presents a Systematic Literature Review (SLR) identifying predominant practices related to MBT suite maintenance, with a focus on strategies for reducing test case discard. The findings reveal that while reuse can prevent the loss of valuable test information, it is fundamentally driven by changes in requirement models, hinging on syntactic differences and the specific constructs of each model's formalism, highlighting the critical role of semantic deepening to minimize test case discard in Test Case Maintenance (TCM) with MBT approaches.

This research full paper presents the perception of programming teachers and students in the context of neurodiverse students. Among the wide range of disabilities, N eurode-velopmental Disorders encompass types of disabilities that are not easily visible, such as Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD). A section of the literature does not consider these disorders to be disabilities. Therefore, neurodiverse describes a person whose brain functions differently than typical. Given this, we can look at these students as people who learn differently and who need to have their specific needs met. With the inclusion of students with disabili-ties in technical and undergraduate courses, they face several challenges, both for these students and teachers. Specifically, several peculiarities can impact this process regarding teaching programming, such as the creation of algorithms based on problems described in extensive texts. Despite efforts to meet the needs of students with disabilities in teaching programming, we still do not have enough studies that address this issue. The main objective of this research is to investigate the current reality of teaching programming to neurodiverse people, what the difficulties are, and how teachers and students are trying to solve them. Thus, given the phenomenon's complexity, we created a case study with two units of analysis: (1) neurodiverse students and (2) their respective programming teachers. The phenomenon studied was programming teaching, and the context was the first programming discipline of two undergraduate courses and a high school technical course. We carried out semi-structured interviews with six students and three teachers. We identified through research that teachers need adequate training to serve atypical students and that educational institutions still need an efficient inclusion process for these students. When we observe the students' vision, we realize that for them to learn to program, there are essential demands, such as the difficulty in understanding the application of programming concepts in practice and the creation of programs without assistance. This study is necessary to show the demand for new approaches to meet the specific needs of neurodiverse students in programming disciplines. To achieve this, inclusive pedagogical strategies and resources must be created and adapted to the characteristics and abilities of these students. Initiatives like this can promote equal opportunities in the classroom, including accessible learning environments, use of explicit language, and individualized support, among others.

This Research Full Paper investigates the use of software testing in introductory programming courses (CS1 and/or CS2) as a supportive and facilitating technique for novice students, identifying adopted strategies and tools used. The literature commonly emphasizes problem-solving skills in programming education or strategies for intervening in educational practices. This paper conducts a Systematic Literature Review (SLR) and aims to understand the impact of introducing tests on novice students' comprehension of programming problems. More research is needed to understand how to improve programming teaching materials and methodologies and how novice students absorb all of this. A study was conducted to answer the following research questions: (RQ1) How are tests being used to facilitate novice students' understanding of programming problems? Additionally, (RQ2) What strategies and tools are being adopted to incorporate software testing into teaching programming for beginner students? The responses highlighted the challenges faced in teaching CS1 and/or CS2, evidencing controlled experiments, alternative methods, and tools used. Finally, we critically analyzed the impact on this target audience and found through the research that there is no consensus on the software testing skills necessary to learn programming. Therefore, these skills were grouped as follows: (1) skills related to introducing software testing tools - consolidated or new - and (2) skills related to education with testing activities. By cross-referencing the information, improvements in student performance can be seen when software tests are integrated into the learning process. Introducing tests by customizing materials adopted in programming courses, activities, and evaluations can promote more effective learning for this group.

This research-to-practice full paper describes a study that employs the CTProgER educational process [37] within High School environments to enhance programming skills. Computational Thinking (CT) is recognized as crucial for developing these skills, especially in high school programming education. Teaching programming presents challenges, necessitating improved educational processes and guidelines. Based on the Anthropological Theory of Didactics, CTProgER operates on the principle that human-created entities are interconnected with individuals, forming relationships denoted as R(X,O). This research investigates the efficacy of CTProgER through a rigorous intervention study conducted in a Brazilian Technical and Vocational High School (TV High School). The results reveal a significant performance disparity between the experimental and control groups, with the experimental group showing notable improvements. These findings highlight the potential of CTProgER to guide and enhance programming instruction within high school curricula, offering valuable insights for validating educational processes.

This Full Research Paper aims to identify and compare the skills involved in learning programming and the difficulties that students with cognitive disabilities may face, discussing the impacts generated by these skills for these students. Several studies talk about what skills a person needs to learn to program. However, we do not have enough to consider the needs of students with disabilities, especially cognitive impairments. Did we conduct a study to answer the following research questions: RQ1 - What skills are involved in programming teaching? RQ2 -Can cognitive impairments affect the skills needed to program? We lift the part the skills involved in learning programming and the cognitive skills impediments in the literature. Finally, we made a critical analysis impact on that audience. By cross-referencing this information, we can highlight that the skills needed to program impact students with cognitive impairments such as math, logic, and problem-solving. This impact does not necessarily mean that these students cannot learn to program. We believe adapting materials, exercises and assessments can foster learning for this audience.

This Research Full Paper presents a Systematic Literature Review (SLR) investigating predictive programming skills and strategies to foster and measure such skills. Predictive skills are specific skills that precede a milestone or the development of other more structured skills, in this case, skills that precede programming skills. Introductory Programming Courses (CS1) feature students with a wide range of skill levels. This difference causes educators to get lost when formulating their practices to teach such a diverse group. There is no clear vision about what previous skills professors should foster before a student enters a CS1, much less which strategies to foster/measure such skills. Due to this limitation, we present/plan an RSL following the guidelines proposed by Kitchenham (2004) to achieve these goals. Our main results are a) Predictive programming skills are problem-solving, abstract thinking, mathematical reasoning, and cognitive flexibility; b) Different researchers use different approaches to teach programming skills based on different educational theories, teaching frameworks, or educational approaches; c) Several studies use the Classical Test Theory as a way to measure predictive programming skills. However, some universities have adopted other theories for this practice, such as Item Response Theory.

This Research-to-Practice Full Paper aims to introduce a methodological framework to foster Computational Thinking (CT) skills by teaching programming with Educational Robotics (ER) in High School. Problem-solving is an essential skill for human development, so it is necessary to introduce actions to improve this skill in High School. It is possible to stimulate problem- solving through computer programming teaching with ER. However, as far as we know, there are no methodologies for teaching programming with ER focusing on CT development in High School that went through scientific validation of the proposal. This study is an Educational Design Research that aims to introduce a methodological framework for teaching programming in High School with ER (named CTProgER) and validate two instances of the framework. It was considered consolidated teaching methodologies with ER, combined with the Anthropological Theory of Didactics and evidence from our ER studies. As a result, we present the CTProgER Framework guideline, two instances of the CTProgER Framework, and these instances’ validation through the Delphi method.

This Research Full Paper presents a Systematic Literature Review (SLR) that analyzes scientific studies based on teaching programming to people with cognitive disabilities. Teaching programming is naturally challenging, especially when faced with students with different learning levels, including cognitive impairment. Due to the lack of a panorama that contemplates the association between teaching programming and people with cognitive disabilities, we present this SLR to identify which initiatives are available to teach people with cognitive disabilities to programming. This SLR followed the protocol model of a systematic review adapted to cover specific requirements for planning, executing, and presenting results. There are some initiatives to include students with some cognitive impairment found in the literature. However, there are still gaps to explore. The main question answered by this research is to identify what are the efforts and contributions to teaching people with cognitive disabilities to program, looking for related papers in the Springerlink, ACM, and IEEE databases.

This Research-to-Practice Full Paper examines the Bebras Challenge tasks, a Computational Thinking (CT) contest that does not require programming knowledge. Numerous studies in the last decade shed light on disseminating CT practices. However, there is still a gap in understanding CT as a construct, that is, CT as a complex cognitive dimension. Furthermore, few studies analyze how many constructs can be recognized inside CT by empirical data. Moreover, algorithmic thinking is one of the essential skills in CT, even when we do not apply programming activities. In this exploratory paper, we investigate the Bebras Challenge tasks. Our primary aim is to figure out how many and which are the constructs associated with CT skills derived from data, especially related to algorithmic thinking. The results point out three primary constructs supported by data. We claim that these constructs are a mix of CT skills, mainly algorithmic thinking, abstraction, generalization, evaluation, and pattern recognition. In addition, our study provides additional evidence supporting the idea that algorithmic thinking can be explored by different perspectives beyond “a set of order instructions”, even without programming practices. Thus, this result reinforces the notion that CT can be understood as a mix of several skills to solve problems. Moreover, algorithmic thinking is a complex cognitive process of thought which may be explored through different strategies. We believe that these findings may help recognize which constructs and abilities underline CT, primarily algorithmic thinking without programming activities.

Educational Robotics (ER) is a tool that can favor teamwork, stimulate creativity, and the ability to solve students' problems through the process of assembling and programming robots. In this sense, ER has been used to improve educational practices. There are several initiatives that seek to stimulate Computational Thinking (CT) to stand out. Although ER being used to develop CT, there are still no guidelines based on scientific studies that support this use. The objective of this Research Full Paper is to identify: a) the main pedagogical approaches are applied in classes with ER focused on developing students' CT; b) the central difficulties are encountered in classes with ER focused on developing students' CT; c) the CT skills are presents in classes with ER focused on developing students' CT; d) the instruments are used to assess students' CT during classes with ER; e) the students' profile who participated in classes with ER focused o on developing students' CT; f) the educational robotics technologies are adopted in classes focused on developing students' CT. To achieve these goals, we planned and carried out systematic mapping of the literature. The results show that, through ER, CT is approached through the teaching of programming languages in a collaborative way in Early Childhood and High School. The most CT skills observed are related to programming, such as sequencing, algorithm, loop, generalization, decomposition, and debugging. The main instruments for assessing students are tests, and the most significant difficulty is the limitation of class hours. Finally, LEGO® is the most used technology in class.

This Research-to-practice Full Paper presents an exploratory and preliminary investigation on the reliability and validity of instruments for measuring introductory programming skills. Our data set consists of the performance of 30 students who participated in the experiment of a Brazilian university. We provide participants with instructional material, practical problems and solutions based on different experimental conditions. The results suggest that the instruments have a good internal consistency index and items with excellent psychometric properties. In addition, some initial evidence to suggest practicing each of the four programming skills. We found that participants who received practice in all four skills obtained a better estimate of study material and Assessment, especially for the most advanced knowledge of programming introduction.

This Research Full Paper presents a Systematic Literature Review (SLR) that investigates state-of-the-art methods, processes, approaches, and instruments based on Item Response Theory (IRT) in the programming teaching context. Various studies support professors and students to improve the evaluation process in teaching programming. Among the different techniques and theories associated with these studies, the IRT gained prominence because of its effectiveness. Due to the lack of an overview in the area, we present a SLR with the objective of identity studies that use IRT as a methodology of evaluation of learning in the context of Programming Teaching; studies which present real-time feedback; and experimental studies that have been carried out to validate them. To achieve these goals, we planned an SLR following the guidelines proposed by Kitchenham (2004). Our main findings are a) There is a limited number of studies that explore IRT in evaluation methodologies in the teaching programming context. Among these studies, the main focus was the development on of instruments to measure programming skills; b) Most instruments are multiple-choice, adopt the 3PL model, and do not show evidence of real-time feedback except SIETTE and CodeWorkout and are also the only instruments that evaluate the coding capacity of individuals; c) The studies showed scientific evidence on the use of IRT in the evaluation methodology in the programming teaching context. The studies presented experiments that indicate a significant gain in the measurement of programming abilities when compared to the traditional evaluation method.

Summary To deliver reliable software, developers may rely on the fault detection capability of test suites. To evaluate this capability, they can apply code coverage metrics before a software release. However, recent research results have shown that these metrics may not provide a solid basis for this evaluation. Moreover, the fixing of a fault has a cost, and not all faults have the same impact regarding software reliability. In this sense, operational testing aims at assessing parts of the system that are more valuable for users. The goal of this work is to investigate whether traditional code coverage and code coverage merged with operational information can be related to post‐release bug detection. We focus on the scope of proprietary software under continuous delivery. We performed an exploratory case study where code branch and statement coverage metrics were collected for each version of a proprietary software together with real usage data of the system. We then measured the ability to explain the bug‐fixing activity after version release using code coverage levels. We found that traditional statement coverage has a moderate negative correlation with bug‐fixing activities, whereas statement coverage merged with the operational profile has a large negative correlation with higher confidence. Developers can consider operational information as an important factor of influence that should be analysed, among other factors, together with code coverage to assess the fault detection capability of a test suite.

This Research Full Paper presents a study on the evaluation of feedback tools in introductory programming classes. Recently, several tools have been proposed in order to provide guidance and help students overcome conceptual difficulties in programming education. Some tools leverage clustering algorithms and program repair techniques to automatically generate personalized hints for students' incorrect programs. In contrast, some teachers choose to present students with program visualization tools to help them understand the dynamic execution of a source code. These tools are used to help students get correct solutions for programming assignments. However, due to limitations in assessments, it is still unclear how effective the feedback provided by these tools is. In this study, we analyzed the effectiveness of a tool for generating personalized hints and a tool for visualizing programs. To do so, we conducted a user study in which students, assisted by these tools, implemented solutions for three programming problems. Our results show that personalized hints can significantly reduce student's effort to get correct solutions. In addition, personalized hints can provide students with an understanding of problem solving similar to when using test cases. However, students who used the program visualization tool got lower post-test performance than using other tools.

This Research to Practice Full Paper investigates the effect of Computational Thinking (CT) and robotics on learning mathematics. CT involves problem-solving skills based on Computer Science concepts. However, there is no CT formal definition as well as information on how to integrate it with other high school subjects. Studies are still searching for methods, tools, and instruments to evaluate CT skills to allow effective adoption. Problem-solving is critical to mathematics, and introducing robotics in this context can favor the learning process. The goal of this study is to apply educational robotics in a Brazilian public high school and evaluate the impact on the development of CT skills and mathematical performance. Two research questions guide the study: (RQ1) Can the use of robotics favor the development of CT skills? (RQ2) Can the use of robotics favor math learning? The development of CT skills was evaluated through the Bebras Challenge, and the annual performance obtained by the students in mathematics was considered to evaluate the impact on mathematics. The overall results pointed out that the use of robotics can favor the learning of mathematics and they bring some indications that the use of robot kits can stimulate the development of CT skills, but still more research is needed.

While several approaches have been proposed to assess computational thinking (CT) abilities, it is still unclear how many and which these abilities are. Despite the undisputed importance of assessment, the fact is that there is not enough evidence on which abilities are merely theoretical and which can be empirically observed in the context of CT. This paper is part of a larger investigation in which we try to answer a simple question: can CT abilities be quantitatively defined and measured? In this particular study, we try to answer a simple question: How many CT abilities can be empirically observed using factor analysis? We approach this question, using a dataset containing answers of 1564 students from Lithuania in the Bebras Challenge from 2015. Firstly, we used confirmatory factor analysis to verify a theory that claims that five CT abilities are assessed by the contest. Our analysis shows the theory is not statistically supported. Secondly, we used principal component analysis as an exploratory analysis to try to derive an appropriate number of factors from the data. Surprisingly, the analysis suggests there are only two main recognizable factors. Finally, we briefly discuss these factors and hypothesize that the first is called evaluation ability, which would include abstraction, generalization, and decomposition, while factor 2 is algorithmic thinking and logical reasoning.

Educational Robotics (ER) has revealed several benefits in the educational context, not only helping the teaching of disciplines, but also making possible the development of several abilities, such as teamwork, problem-solving, and creativity. Among various robotics kits, LEGO®Robotics has been shown one of the best results considering some evaluated criteria (modularity level, hardware, curriculum, price, etc.). Some studies analyze the teaching practices, some compare technologies, and others evaluate the kits in a pedagogical way. However, it is essential to investigate all these contexts together in order to improve the impact produced by the ER in education and to know the best teaching practices associated with the most powerful technologies. The objective of this Research Full Paper is to identify: a) environments and programming languages adopted in the LEGO®Robotics context, b) educational practices applied during classes based on LEGO®Robotics, and c) the educational levels in which robotics has been applied with positive results. To achieve these goals, we planned and carried out a systematic review of the literature. Our main findings are: a) the most widely used environment and programming language are LabVIEW along with the LEGO®'s block-based programming language, b) we identified LEGO®Robotics is used for teaching programming, interdisciplinary contents, participation in tournaments, robotics, and computational thinking, c) LEGO®Robotics is used with success by students of different levels, such as K12, undergraduate, and graduated. Finally, we discuss some problems and limitations related to ER and point out that there is no standardization of teaching practices or methodologies for evaluating results, indicating that more research is needed to find the best scenario regarding technologies, methods, and target audience.

In agile development, Requirements Engineering (RE) and testing have to cope with a number of challenges such as continuous requirement changes and the need for minimal and manageable documentation. In this sense, extensive research has been conducted to automatically generate test cases from (structured) natural language documents using Model-Based Testing (MBT). However, the imposed structure may impair agile practices or test case generation. In this paper, inspired by cooperation with industry partners, we propose CLARET, a notation that allows the creation of use case specifications using natural language to be used as central artifacts for both RE and MBT practices. A tool set supports CLARET specification by checking syntax of use cases structure as well as providing visualization of flows for use case revisions. We also present exploratory studies on the use of CLARET to create RE documents as well as on their use as part of a system testing process based on MBT. Results show that, with CLARET, we can document use cases in a cost-effective way. Moreover, a survey with professional developers shows that CLARET use cases are easy to read and write. Furthermore, CLARET has been successfully applied during specification, development and testing of industrial applications.

Several instruments have been used for assessing Computational Thinking (CT) abilities. In this exploratory and preliminary study, we investigate how appropriate the Bebras challenge is as an instrument to assess and measure CT abilities. Bebras is an international challenge whose goal is to promote Computer Science and CT. The test can be answered without any prior knowledge on computer science. Our broad research question is whether we can evolve Bebras into a full fledged instrument to assess and measure CT abilities. In this paper, we instantiate a few more specific research questions: Is Bebras performance a good predictor of success for students within programming courses? Is there any correlation between Bebras performance and students' grades? Do students improve their performance in Bebras tests when exposed to the contents of a programming course? Our dataset consists of the grades of 138 students who attended introductory programming courses at two Brazilian universities and their performance in two simulated Bebras tests. The first test was applied at the beginning of the term and therefore before any exposition to programming classes. The second one was applied at the end of the term. The results suggest that the performance on Bebras is only moderately correlated to the student grades. We conclude that it is not very likely that CT measures can be derived from the Bebras test as it is currently designed. While further research is needed on how we can leverage the Bebras effort to extend it into a CT assessment instrument, we performed a preliminary study on the use of Item Response Theory (IRT) as a means to improve the selection of questions and the design of the test. We expect the results of this research can contribute both to the development and discussion on CT assessment as well as to the Bebras effort to educate CT.

Several initiatives have been created to promote Computational Thinking (CT) abilities in students. There are multiple approaches of assessing CT and wide abilities and skills involved. However, the evidence on how to assess CT has not yet been systematically grouped or reviewed. The goal of our study is to identify and classify approaches to promote CT and the different ways of assessing CT abilities. To achieve this goal, a systematic mapping study was planned and executed. The results reveal that: (i) programming courses are the most common pedagogical approaches to promote CT for K-12 students; (ii) multiple skills are involved in CT, but solving problems, algorithms, and abstraction are most common abilities assessed; and (iii) codes and multi-choice questionnaires are the most common artifacts for assessing CT abilities. This study points out to the fact that there are open questions for exploring and developing new researches for promoting and assessing CT abilities.

Some researchers consider that the benefits of Computational Thinking (CT) in education are unclear, and therefore a quantitative study to understand the effects of CT skills in the basic levels of formal education is needed. The present study aims to analyze the effects of CT on the academic performance of students in high school. We seek to answer the following research questions: (RQ1) Can CT increase the problem-solving skills of students in high school? (RQ2) Is CT related to the performance of students in high school? (RQ3) Can CT improve the performance of students in high school? We conducted an analysis between two groups of students with similar profiles belonging to the same population, where the factor CT proficiency is the only feature that distinguishes both groups. The results show that there is a moderate correlation between the performance of evaluated students and their skills of CT developed by computer programming. In addition, we found that students proficient in computer programming have superior performance up to 10.2% in ENEM (National Exam of High School) assessment compared to those with little or no experience level and even better performance 21% in the WASI test problem-solving in students on attending high school.

Real-time systems are composed of subsystems that may communicate by means of interruptions. An interruption is an event that requires preemption of a resource held by an executing subsystem. This subsystem may resume its execution from the point where it stopped when interruption handling finishes its execution. Testing systems composed of interruptions is hard since interruptions may happen at different points of execution. This fact demands the tester to apply a systematic procedure to specify and perform testing of the interruption behavior at specific points of interest. In this paper, we propose a symbolic approach to generating interruption test cases from real-time system models. We define an interruption operator for a timed model along with its properties. An empirical study illustrates its applicability to test application level interruptions on an Android system.

The state space explosion problem is one of the challenges to be faced by test case generation techniques, particularly when data values need to be enumerated. This problem gets even worse for real-time systems (RTS) that also have time constraints. The usual solution in this context, based on finite state machines or time automata, consists of enumerating data values (restricted to finite domains) while treating time symbolically. In this paper, a symbolic model for conformance testing of real-time systems software named TIOSTS that addresses both data and time symbolically is presented. Moreover, a test case generation process is defined to select more general test cases with variables and parameters that can be instantiated at testing execution time. Generation is based on a combination of symbolic execution and constraint solving for the data part and symbolic analysis for timed aspects. Furthermore, the practical application of the process is investigated through a case study.

Abstract Modelling and testing of reactive systems with interruptions are discussed. These systems are commonly found in portable devices, where interruptions to a running application can be demanded at any time, due to concurrent execution of processes sharing a single resource, such as screen, as well as arrival of calls from network distributed services. Since the possible number of combinations of allowed interruptions is large, proper test case selection activities need to be performed. But, in order to systematically investigate and select test cases, it is fundamental to explicitly model interruption behaviour in a compositional way, avoiding the need for explicit enumeration. This work presents a strategy for testing interruptions in reactive systems that covers modelling for testing of systems with interruptions, generation and selection of sound test cases. The strategy is supported by the LTS-BT tool. Moreover, a formal model of an environment devoted to execution of test cases with interruptions is presented. Finally, a case study illustrates its applicability in the mobile phone application domain.

A method of functional testing for software components according to model-based testing techniques is proposed. Test cases are generated from UML diagrams and OCL constraints that comprise a component interface and realization specification. The method uses a reduced set of UML artifacts that constitute the main requirements for its application along with a component development process, making use of development artifacts. Also, the set of generated test artifacts are packed together with the provided components to reduce the overall testing effort when clients assemble applications. A tool has been developed to automate the method with test cases generated as Java test components. Test execution and result analysis is also supported. For each component, the tool generates a test component that can be easily upgraded and configured for testing the services provided by a component throughout its life cycle.