Conceptual modelling can be used as a starting point for designing software systems and to better understand the domain knowledge related to requirements. This can act as a first step in Model-Driven Engineering (MDE) which is an approach to software development where code (or another artefact such as documentation or test cases) is automatically generated from a model. Assuming that the code or other artefacts are generated correctly from the designed conceptual model by the transformation engine, there is still no guarantee that the model correctly reflects the requirements for the system. This validation can take the form of model simulation, where the user can simulate real-world scenarios on the model. It has been found that students often struggle with modelling. Previous research has found that the use of a model simulation tool can improve student performance in terms of understanding the behavioural and structural aspects of a model and check the output for correctness. However, due to the exploratory nature of simulation tools, it is possible that some patterns of interacting with a model simulation tool are more typical of high achieving students than low achieving students. To this end, this research aims to analyse students' interaction behaviours in a model simulation tool in order to understand which approaches to model validation are most fruitful for good performance in an conceptual modelling course. This is done by collecting students' recorded user logs in a simulation tool and developing a process model using process mining. The results show that highperforming students show different interaction behaviours than average and low-performing students. Several recommendations for teaching conceptual modelling are derived from the identified interaction behaviours.

This paper presents an industrial experience applying random scriptless GUI testing to the Yoho web application developed by Marviq. The study was motivated by several key challenges faced by the company, including the need to optimise testing resources, explore how random testing can complement manual testing, and investigate new coverage metrics, such as “code smell coverage”, to assess software quality and maintainability. We conducted an experiment to explore the impact of the number and length of random GUI test sequences on traditional adequacy metrics, the complementarity of random with manual testing, and the relationship between code smell coverage and traditional code coverage. Using Testar for scriptless testing and SonarQube code smell identification, results show that longer random test sequences yielded better test adequacy metrics and increased code smell coverage. In addition, random testing offers promising efficiency in test coverage and detects unique smells that m anual testing might overlook. Additionally, including code smell coverage provides valuable insights into long-term code maintainability, revealing gaps that traditional metrics may not capture. These findings highlight the benefits of combining functional testing with metrics assessing code quality, particularly in resource-constrained environments.

Scriptless testing generates test sequences dynamically by automatically exploring the Graphical User Interface (GUI). Instead of relying on predefined scripts-which have proven expensive to maintain-scriptless tools detect available widgets, derive possible actions, and select actions on the fly using exploratory techniques such as random selection, model-based inference, or reinforcement learning. While scriptless testing is a valuable complement to scripted approaches, current techniques lack the intelligence needed to strategically select and execute GUI actions that fulfill specific functional testing goals-such as those derived from requirements, use cases, or user stories. Unsurprisingly, this leads companies to question the viability of scriptless testing tools and continue relying on scripts for test automation. This paper reports on the integration of Large Language Models (LLMs) into a scriptless GUI testing tool for action selection, aiming to determine whether it can generate effective action sequences to test specific functional requirements. Our results demonstrate that a multi-objective test goal structure, combined with historical and feedback context, enables LLM-empowered scriptless testing to automate functional testing. Although further research is needed to tackle challenges in complex test scenarios, our findings offer promising results that LLM-empowered scriptless testing can reduce reliance on the maintenance-heavy aspects of traditional scripted testing.

Computer 3D games are complex software environments that require novel testing processes to ensure high-quality standards. The Intelligent Verification/Validation for Extended Reality Based Systems ( iv4XR ) framework addresses this need by enabling the implementation of autonomous agents to automate game testing scenarios. This framework facilitates the automation of regression test cases for complex 3D games like Space Engineers. Nevertheless, the technical expertise required to define test scripts using iv4XR can constrain seamless collaboration between developers and testers. This paper reports how integrating a Behavior-Driven Development (BDD) approach with the iv4XR framework allows the industrial company behind Space Engineers to automate regression testing. The success of this industrial collaboration has inspired the iv4XR team to integrate the BDD approach to improve the automation of play-testing for the experimental 3D game LabRecruits. Furthermore, the iv4XR framework has been extended with tactical programming to enable the automation of long-play test scenarios in Space Engineers. These results underscore the versatility of the iv4XR framework in supporting diverse testing approaches while showcasing how BDD empowers users to create, manage, and execute automated game tests using comprehensive and human-readable statements.

Scriptless testing at the GUI level involves generating test sequences on the fly. These test sequences mimic user interactions on the GUI. The creation of these sequences works through action selection rules, which is most commonly based on stochastic methods. Script-less tests are reliable because they work with the actual state of the System Under Test (SUT). However, the tests are less specific, harder to interpret, and it is difficult to test concrete use cases or workflows. We want to tackle this drawback of scriptless tests by introducing action selection rules that are easier to guide than pure stochastic methods. In this paper, a new approach based on a grammar for the action selection rules is proposed, enabling scriptless testing tools to mimic user behaviour when interacting with web applications. While grammars have been used in software testing to generate input data for test cases, the proposed approach uses grammars to specify action selection rules to generate test sequences that mimic testing strategies employed by human testers. An empirical study has been performed to evaluate the effectiveness and the efficiency of the grammar-based action selection rules to filing web forms in comparison with random action selection rules. In the study, two SUTs were used: WebformSUT and Parabank. The average success rate for the grammar-based approach was 95.9% against random's 57.0% for WebformSUT and 99.8% against 55.7% for Parabank. For the widget interaction grammar-based had an average deviation from the ideal ratio of 0.06165 (WebformSUT) and 0.0180 (Parabank), compared random's 0.4318 (WebformSUT) and 0.7774 (Parabank). The results demonstrate the effectiveness of the grammar-based approach and the improvement in the use of resources.

This study examines sensemaking in student test case design, showing a reliance on conceptual knowledge learned during programming courses over exploratory testing strategies. Of the three identified approaches taken by students, the “developer approach” is used most often, suggesting a gap in software engineering education. We hypothesise that software testing should be taught in computer science programs using a design paradigm based on empiricism instead of rationalism. Based on these results, and our hypothesis, we will further analyse the sensemaking processes of both students and experts, and create an instructional design to improve software testing education in computer science programs.

Computer games have reached unprecedented importance, exceeding two billion users in the early 2020s. Human game testers bring invaluable expertise to evaluate complex games like 3D sandbox games. However, the sheer scale and diversity of game content constrain their ability to explore all scenarios manually. Recognizing the significance and inherent complexity of game testing, our research aims to investigate new automated testing approaches. To achieve this goal, we have integrated scriptless testing into the industrial game Space Engineers, enabling an automated approach to explore and test sandbox game scenarios. Our approach involves the development of a Space Engineers-plugin, leveraging the Intelligent Verification and Validation for Extended Reality-Based Systems (IV4XR) framework and extending the capabilities of the open-source scriptless testing tool TESTAR. Through this research, we unveil the potential of a scriptless agent to explore 3D sandbox game scenarios autonomously. Results demonstrate the effectiveness of an autonomous scriptless agent in achieving spatial coverage when exploring and (dis)covering elements within the 3D sandbox game.

Software testing is an indispensable component of software development, yet it often receives insufficient attention. The lack of a robust testing culture within computer science and informatics curricula contributes to a shortage of testing expertise in the software industry. Addressing this problem at its root -education- is paramount. In this paper, we conduct a comprehensive mapping review of software testing courses, elucidating their core attributes and shedding light on prevalent subjects and instructional methodologies. We mapped 117 courses offered by Computer Science (and related) degrees in 49 academic institutions from four Western European countries, namely Belgium, Italy, Portugal and Spain. The testing subjects were mapped against the conceptual framework provided by the ISO/IEC/IEEE 29119 standard on software testing. Among the results, the study showed that dedicated software testing courses are offered by only 39% of the analysed universities, whereas the basics of software testing are taught in at least one course at every university. The analysis of the software testing topics highlights the gaps that need to be filled in order to better align the current academic offerings with the real industry needs.

Many educators face problems with integrating testing into programming education. For instance: existing courses are already fully packed; testing requires skills that students might not yet have; and testing is, although considered important, not always given priority by students. Educators, in general, do not have time to overhaul a programming course to fully integrate testing, resulting in a situation in which the improvement of testing education seems to have slowed down. In this paper, we propose Test Informed Learning with Examples (TILE), a new concept to create test-awareness in introductory programming courses. TILE aims to introduce testing as early as possible and in a subtle way. As a result, integration into existing curricula can be done seamlessly and requires less effort than completely overhauling existing programming courses. The contributions of this paper are: the presentation of TILE; experiences of having applied this method in the classroom; and an open repository with assignments using our approach. Applying TILE seems to be a promising approach to introduce testing in early programming. Moreover, some TILEs can be added to existing courses with almost no effort from day one. More research is needed to gain confidence in the benefits of using TILE over time and to collect evidence that we reached the final aim of TILE, i.e. students that test because that inherently belongs to programming, and not because it is explicitly asked from them.

Software testing is the most used technique for quality assurance in industry. However, in computer science education software testing is still treated as a second-class citizen and students are unable to test their software well enough. One reason for this is that teaching the subject of software testing is difficult as it is a complex intellectual activity for which students need to allocate multiple cognitive resources at the same time. A myriad of primary and secondary studies have tried to solve this problem in education, however still with very limited results. Before we can design interventions to improve our pedagogical approaches, we need to gain more in-depth understanding and recognition of sensemaking as it is happening when students design test cases. An initial exploratory study identified four different sensemaking approaches used by students while creating test models. In this paper we present a follow-up study with 50 students from a large university in Spain. The used methodology was based on the previous study with the improvements that originated from its evaluation. We asked the participants to create a test model based on a description of a test problem using a specialized web-based tool for modeling test cases. We measured how well these models fit the test problem, the sensemaking process that students went through when creating the models, and the students’ perception of the modeling task. The participants received no compensation for their efforts, and we scheduled the experiment during a regular class. Apart from the created models and their metadata, we also collected recordings of the students’ computer screens made during the experiment and used a questionnaire to study their perspectives on the assignment. All the collected textual, graphical, and video data was analyzed using an iterative inductive analysis process to allow new information about the different sensemaking approaches to emerge. We gained better insights into the sensemaking processes of students while modeling test cases for a problem. The results enabled us to refine our previous findings, and we identified new sensemaking approaches. Based on these results, we can further investigate ways to influence the sensemaking process in education, the possible misconceptions that have a negative influence on it, and the desired mental model we want our students to have to design test cases.

Automation is required in the software development to reduce the high costs of producing software and to address the short release cycles of modern development processes. Lot of effort has been performed to automate testing, which is one of the most resource-consuming development phases. Automation of testing through the Graphical User Interface (GUI) has been researched to improve the system testing. We aim to evaluate the complementarity of automated GUI testing tools in a real industrial context, which refers to the capability of the tools to work usefully together. To address the objective, we conduct an exploratory case study in an IT development company from The Netherlands. We select two representative tools for automated GUI testing, one for scripted and another for scriptless testing. We measure the complementarity by measuring the effectiveness, the efficiency, and subjective satisfaction of the tools. It can be observed that the scripted tool performs better in detecting process failures, and the scriptless tool performs better in detecting visible failures and also reaching higher coverage. Both tools perform in a similar way in terms of efficiency. Additionally, both tools were perceived to be useful in the survey performed for the subjective satisfaction. We conclude that scriptless and scripted testing approaches are complementary, and they can improve the effectiveness compared to manual testing processes performed in an industrial context by detecting different failures and reducing the effort and time to find these failures and to reproduce them.

State model inference of software applications through the Graphical User Interface (GUI) is a technique that identifies GUI states and transitions, and maps them into a model. Scriptless GUI testing tools can benefit substantially from the availability of these state models, for example, to improve the exploration, or have sophisticated test oracles. However, inferring models for large systems requires a long execution time. Our goal is to improve the speed of the state model inference process. To achieve this goal, this paper presents a distributed state model inference approach with an open source scriptless GUI testing tool. Moreover, in order to be able to infer a suitable model, we design a set of strategies to deal with abstraction challenges and to distinguish GUI states and transitions in the model. To validate it, we conduct an experiment with two open source web applications that have been tested with the distributed architecture using one to six Docker containers sharing the same state model. With the obtained results, we can conclude that it is feasible to infer a model with a distributed approach and that using the distributed approach reduces the time required for inferring a state model.

Context: There is a lack of empirical evidence on the differences between model-driven development (MDD), where code is automatically derived from conceptual models, and traditional software development method, where code is manually written. In our previous work, we compared both methods in a baseline experiment concluding that quality of the software developed following MDD was significantly better only for more complex problems (with more function points). Quality was measured through test cases run on a functional system. Objective: This paper reports six replications of the baseline to study the impact of problem complexity on software quality in the context of MDD. Method: We conducted replications of two types: strict replications and object replications. Strict replications were similar to the baseline, whereas we used more complex experimental objects (problems) in the object replications. Results: MDD yields better quality independently of problem complexity with a moderate effect size. This effect is bigger for problems that are more complex. Conclusions: Thanks to the bigger size of the sample after aggregating replications, we discovered an effect that the baseline had not revealed due to the small sample size. The baseline results hold, which suggests that MDD yields better quality for more complex problems.

Context: The testing phase is an important but very expensive part of development process. Model-Based Testing (MBT) allows the automation of the generation of test cases to decrease part of this high cost by using software conceptual models. Indeed, JMermaid is a tool that allows the automatic generation of software applications from the conceptual models. Objective: The main objective of this investigation is to develop a module to visualize the coverage of test cases of the applications generated by the JMermaid tool. Method: Model-based testing technique was used to generate test cases from the JMermaid conceptual models, and a tool called TesCaV was implemented for the automatic generation of test cases. Results: TesCaV was integrated with JMermaid allowing the visualization of the coverage of test cases on the generated Java applications. Conclusions: The visualization of test cases improves the understanding of the testing phase in the development process.

Context: Programming courses are compulsory for most engineering degrees, but students’ performance on these courses is often not as good as expected. Programming is difficult for students to learn, given that it includes a lot of new, complex, and abstract topics. All of this has led experts to the conclusion that new teaching techniques are required if students are to be motivated and engaged in learning on programming courses. Gamification has come to be an effective technique in education in general, and is especially useful in programming courses. This motivated us to develop an open source gamified platform, called UDPiler, for use in a programming course. Objective: The main goal of this article is to obtain empirical evidence on the improvement of students’ learning performance when using UDPiler in comparison to a non-gamified compiler. Method: A quasi-experiment was performed with two groups of first-year engineering students at Diego Portales University in Chile, using a non-gamified compiler and a gamified platform, respectively. Results: The results reveal that the students obtained better marks when the gamified platform was used to learn C programming. In addition, there is statistical significance in favor of there being a positive effect on the learning performance of those students who used the gamified platform. Conclusions: The results allow us to conclude that gamification is an encouraging approach with which to teach C programming, a finding that is aligned with previous empirical studies concerning gamification on programming courses, carried out in academic contexts. Nonetheless, we are aware that further validation is also required to corroborate and strengthen the findings obtained and to investigate whether the kind of gamified elements (mechanics, dynamics, and aesthetics) used have any influence on students’ performance, among other issues that deserve further investigation and that are explained throughout this article.

One of the more complex topics to teach to software engineering students is the conceptual modeling design, which has several concepts that students must learn in order to specify the structural, behavioral and interaction views of software systems.Learning the design of class diagrams is of paramount importance since these diagrams are used to guide concrete development tasks such as programming and software testing, and -consequently-to avoid defective software products.Applying novel teaching/learning techniques in this topic may help students to reduce the defects that are committed at the moment of designing a class diagram.One interesting technique is the use of serious games, due to the fact that they provide learning environments free of risks and pressure for students, allowing the students to know the topics that they must learn in a fun way.Serious games have been widely used in programming courses.Thus, we aim to investigate the feasibility to replicate this experience for conceptual modeling of class diagrams at software engineering courses.In this paper, we present a roleplaying game specially focused in the class diagram, which is called Classutopia.This serious game provides modeling challenges, comprehension and correction of diagrams with different complexity levels for learning conceptual modeling design.

One of the more complex topics to teach to Software Engineering students is the conceptual modeling design, which has several concepts that students must learn in order to specify the structural, behavioral and interaction views of software systems. Learning the design of class diagrams is of paramount importance since these diagrams are used to guide concrete development tasks such as programming and testing, and, consequently, to avoid defective software products. Applying novel teaching/learning techniques in this topic may help students to reduce the defects that are committed at the moment of designing a class diagram. One interesting technique is the use of serious games, since they provide learning environments free of risks and pressure for students, allowing the students to know the topics that they must learn in a funny way. Serious games have been widely used in programming courses. We aim to investigate the feasibility to replicate this experience for conceptual modeling of class diagrams at Software Engineering courses. In this paper, we present a role-playing game especially focused on the class diagram, which is called Classutopia. This serious game provides modeling challenges, comprehension and correction of diagrams with different complexity levels for learning conceptual modeling design.

Context. Many critical systems must meet safety compliance needs from safety standards. These standards are usually large textual documents whose compliance needs can be hard to understand. As a solution, the use of models has been proposed. Goal. We aim to provide evidence of the extent to which models improve the understanding of safety compliance needs. Method. We designed an experiment and ran a pilot to study the effectiveness, efficiency, and perceived benefits of understanding these needs, with the text of standards and with models in the form of UML object diagrams. Results. The overall results from 15 Bachelor students show that the effectiveness of understanding safety compliance needs increases very little with models (2%), and the efficiency even decreases (24%). Nonetheless, the results improve when the potential complexity in navigating the models is taken into account (15% effectiveness increase). The students find benefits in using the models but most consider that the models are hard to understand. Conclusions. The extent to which models improve the understanding of safety compliance needs seems to be lower than what the research community expects. New studies are necessary to confirm our initial insights.

The market of mobile applications is constantly changing and growing, partly due to the popularity of software that supports daily tasks and the emergence of new communication medias. Applications of this kind are typically small but require high usability interfaces, which are often based on small's sets of patterns with proven efficacy. Using patterns is one of the main principles of model-driven development, a paradigm with the potential to reduce development time and coding errors. However, there exist many challenges for improving the usability of automatically generated mobile interfaces. To face some of these challenges, this paper proposes a method to model mobile interfaces, which is part of an ongoing research project aimed at model-driven development of fully-functional mobile applications. We present the modeling constructs, which are inspired by recent studies on best practices, and evaluate the feasibility of the application of the proposal to an archetypical case.

The main focus of Software Engineering discipline is to establishing methods and processes for the effective and efficient development of software projects. One of the most relevant paradigms for achieving this goal is Model-Driven Development (MDD). MDD uses models at different abstraction levels to automatically generate software products by means of model-transformations. However, one of the main issues related to the development of MDD technologies is the lack of standardization in terms of the features that need to be considered to support the current industry needs. This difficults the comparison of existing technologies since there are not a reference point for the creation of new MDD approaches with their corresponding supporting tools. This paper analyses these industry needs through an exploratory study. From the results obtained, it states the main features that must be supported by MDD tools. In addition, this paper presents an analysis about the presence or absence of these features in a set of industrial MDD tools.

Models are key artifacts in Model-Driven Development (MDD) methods. To produce high-quality software by using MDD methods, quality assurance of models is of paramount importance. To evaluate the quality of models, defect detection is considered a suitable approach and is usually applied using reading techniques. However, these reading techniques have limitations and constraints, and new techniques are required to improve the efficiency at finding as many defects as possible. This article presents a case study that has been carried out to evaluate the use of a Functional Size Measurement (FSM) procedure in the detection of defects in models of an MDD environment. To do this, we compare the defects and the defect types found by an inspection group with the defects and the defect types found by the FSM procedure. The results indicate that the FSM is useful since it finds all the defects related to a specific defect type, it finds different defect types than an inspection group, and it finds defects related to the correctness and the consistency of the models.

Nowadays, the emergence of several model-driven development (MDD) proposals that are related to multiple domains requires the definition of proper interoperability mechanisms that facilitate the reuse of knowledge in the MDD community by taking advantage of already defined modeling languages, tools, and standards. However, there are no recent studies that cover the existent interoperability alternatives in the model-driven domain nor is there a common interoperability framework. This paper confronts this situation through a systematic analysis of recent interoperability approaches that provide relevant features for MDD processes. From this analysis, a general interoperability framework is depicted, which is complemented with our contributions to solve specific interoperability issues. Therefore, we present those aspects that are already covered by existent proposals as well as those pending subjects that, from our point of view, are future challenges in order to achieve a suitable interoperability framework for MDD approaches.

The current Web applications are in continuous evolution to provide new and more complex functionalities, which can improve the user experience by means of adaptivity and dynamic changes. Since testing is the most frequently used technique to evaluate the quality of software applications in industry, novel testing approaches will be necessary to evaluate the quality of future (and more complex) web applications. In this paper, we investigate the testing challenges of future web applications and propose a testing methodology that addresses these challenges by the integration of search-based testing, model-based testing, oracle learning, concurrency testing, combinatorial testing, regression testing, and coverage analysis. This paper also presents a testing metamodel that states testing concepts and their relationships, which are used as the theoretical basis of the proposed testing methodology.

Applying new learning methodologies in education, such as visual learning based on virtual reality and three-dimensional (3D) environments, is an important aspect in education, since it offers possibilities that can remarkably improve the current education system. Technological advances, along with the chance to create and represent the varying contents offered by information technologies, make the new learning methodologies the focus of attention in the future. Currently, 3D methodologies are only used in Computer Science to improve physical characteristics (virtual laboratories, virtual worlds, etc.), but they are not used to improve the internal mental processes by which human beings understand and retain abstract concepts. In these cases, the use of visual learning helps to clarify them. In Computer Science, particularly in Software measurement courses, the complexity of the concepts is possibly greater than in other courses because there is a lot of learning material that is based on abstract concepts that students find hard to recognize in the real world. In this paper, we present a visual environment that can be used to learn software measurement concepts like the IFPUG functional size measurement method. To validate the new learning model, an experiment was carried out.

Models are key artifacts in Model-Driven Development (MDD) methods. To evaluate the quality of models, defect detection is considered to be a suitable approach, which is usually applied using reading techniques. However, new techniques are required in order to find as many defects as possible. This paper presents a case study to evaluate the usefulness of a Functional Size Measurement (FSM) procedure to detect defects in models of a MDD environment. The results indicate that the FSM is useful in finding all the defects that are related to a defect type as well as finding different defect types than an inspection team does.

A key requirement for MDD solutions is to have a modeling language that allows the correct representation of conceptual models. Nowadays, there are two options that are the most widely used for the definition of these modeling languages: 1) the specification of a domain-specific modeling language (DSML) or 2) the customization of UML. In practice, these two modeling alternatives are viewed as opposite solutions. However, since both alternatives provide benefits for the application of MDD solutions, in this paper, we present a proposal that uses UML profile extension mechanisms to interchange modeling information between DSML-based models and UML models. This proposal shows how these two modeling alternatives can be integrated in a unique MDD solution.