Context: Metamorphic testing (MT) is a well-known approach to address the oracle problem in software testing. It does so by using expected relations – called metamorphic relations (MRs) – between the inputs and outputs of multiple system executions as test oracles. This approach involves identifying meaningful MRs for the domain, defining them in an executable language, and creating suitable system inputs to check their satisfaction. However, these tasks are manual and demand substantial effort from testers. Objective: To reduce the effort of applying MT, we propose automating some tasks in the MT process with intelligent assistants, focussing on MR inference and generation of follow-up test cases. Methods: We propose a taxonomy of MT tasks amenable to LLM-based assistance, and an extensible architecture for their automation. The MT tasks are accessible via a conversational assistant, and may encompass validation and fixing cycles to improve the quality of the assistance. We have integrated the architecture with the MT framework Gotten . Results: We have evaluated our approach on the two considered MT tasks across three domains (data centres, autonomous vehicles, finite automata). We found the assistant effective at generating new MRs (between 2 and 6 depending on the domain) and follow-up test cases (with over 99.6% correctness). We also measured diversity and redundancy of the generated MRs, the benefits of fixing cycles on different LLMs when generating follow-up test cases, and performance degradation and computational cost as input complexity increases. Conclusions: Our architecture effectively automates two central MT tasks, namely, inferring MRs and generating follow-up test cases, showing potential to reduce the effort from MT practitioners and to automate other MT tasks.

Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots – like ChatGPT – can converse on any topic, task-oriented chatbots – the focus of this paper – are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.

Testing is essential to improve the correctness of software systems. Metamorphic testing (MT) is an approach especially suited when the system under test lacks oracles, or they are expensive to compute. However, building an MT environment for a particular domain (e.g., cloud simulation, model transformation, machine learning) requires substantial effort. Our goal is to facilitate the construction of MT environments for specific domains. We propose a model-driven engineering approach to automate the construction of MT environments. Starting from a meta-model capturing the domain concepts, and a description of the domain execution environment, our approach produces an MT environment featuring comprehensive support for the MT process. This includes the definition of domain-specific metamorphic relations, their evaluation, detailed reporting of the testing results, and the automated search-based generation of follow-up test cases. Our method is supported by an extensible platform for Eclipse, called Gotten. We demonstrate its effectiveness by creating an MT environment for simulation-based testing of data centres and comparing with existing tools; its suitability to conduct MT processes by replicating previous experiments; and its generality by building another MT environment for video streaming APIs. Gotten is the first platform targeted at reducing the development effort of domain-specific MT environments. The environments created with Gotten facilitate the specification of metamorphic relations, their evaluation, and the generation of new test cases.

Creating and grading exercises are recurring tasks within higher education. When these exercises are based on diagrams – like logic circuits, automata or class diagrams – we can represent them as models, and use model-driven engineering techniques for the large-scale generation of quizzes, which can be automatically graded. This way, we propose a domain-independent tool for the generation and automated evaluation of diagram-based exercises called Wodel-Edu. Wodel-Edu is built atop Wodel, an extensible tool for model mutation, and offers seven kinds of diagram exercises. It supports code generation from the exercises for the Moodle platform, the web, Android and iOS applications. Evaluations from the professor and student perspectives show good results.

Behavioral models are important assets that must be thoroughly verified early in the design process. This can be achieved with manually-written test cases that embed carefully hand-picked domain-specific input data. However, such test cases may not always reach the desired level of quality, such as high coverage or being able to localize faults efficiently. Test amplification is an interesting emergent approach to improve a test suite by automatically generating new test cases out of existing manually-written ones. Yet, while ad-hoc test amplification solutions have been proposed for a few programming languages, no solution currently exists for amplifying the test cases of behavioral models.

Two crucial aspects for the trustworthy utilization of domain-specific languages (DSLs) are their semantic correctness, and proper testing support for their users. Testing is frequently used to verify correctness, but is often done informally -- which may yield unreliable results -- and requires substantial effort for creating suitable test cases and oracles.

Mutation testing (MT) is a well-known technique to evaluate and improve the quality of a given test-suite. While several MT tools exist for traditional programming languages, there is no common method to take advantage of MT in the case of domain-specific languages (DSLs). The current MT tools for DSLs are created ad-hoc, incurring in a high cost.