Abstract In the digital age, the omnipresence of software drives the mass production fueled by artificial intelligence. This dependence requires rigorous methods to assess the quality of the software. Inspired by Proficiency Tests (PT) from other fields, this work proposes an innovative process to evaluate the competence of software testing teams and laboratories. Based on Action Design Research(ADR) and Business Process Management(BPM), the developed solution includes software that automates the process, provides a knowledge base, and generates performance indicators. Furthermore, we explore the use of Large Language Models (LLMs) to extract valuable insights from PT results. Quantitative experiments demonstrated the effectiveness of the proposed process, while a qualitative study validated the solution, demonstrating its feasibility and potential to transform the quality assessment of software.

Summary Graphic processors offer an accessible solution for high‐performance computing, addressing challenges across various fields. The Compute Unified Device Architecture (CUDA) programming model has emerged to enhance the performance of general‐purpose applications on graphic processors. However, developing CUDA programs is far from straightforward, and developers' lack of experience in parallel programming has led to numerous issues. This article presents a structural testing model and criteria to improve the quality of CUDA programs. These criteria facilitate the selection of test cases and aid in identifying faults. The ValiCUDA tool was developed to implement and validate this testing model and criteria. This tool instruments and analyzes programs, generating the necessary elements for each testing criterion. It also facilitates program execution and evaluation of criterion coverage. A statistical validation experiment assessed these criteria' effectiveness, cost, and strength metrics. The results demonstrate that the criteria can identify nontrivial faults in CUDA programs and assist testers in their testing endeavors for such applications.

This paper presents an investigation of an Experi-ential Learning approach aiming to: (i) provide an experience which bring students closer to practice by means of a real project; and (ii) foster soft skills. Our experience took place in a Software Engineering graduate course, delivered remotely. This work and its results are primarily to be viewed as a feasibility study, although the findings provide interesting results. Active strategies, especially the Experiential Learning, can be a useful approach to provide an experience closer to real-world software industry practices, besides to foster soft skills in the students (particularly Communication, Teamwork, Leadership and Adaptability and Flexibility).

Summary Concurrent programming is increasingly present in modern applications. Although it provides higher performance and better use of available resources, the mechanisms of interaction between processes/threads result in a greater challenge for software testing activity. The nondeterminism present in those applications is one of the main issues during the test activity since the same test input can produce different possible execution paths, which may or not contain defects. The test data automatic generation can alleviate this problem, ensuring higher speed and reliability in software testing activity. This paper explores the automatic test data generation for concurrent programs through Genetic Algorithm, a bioinspired optimization technique, and proposes a test data generation approach for concurrent programs, called BioConcST, and a new operator for the selection of test subjects, called FuzzyST, which uses fuzzy logic. The approaches were evaluated in an experimental study towards their validation. The results showed that BioConcST is more promising than the other approaches at all analyzed levels. FuzzyST, together with Elitism and Tournament operators, provided the best results; however, it proved more suitable for concurrent programs of higher complexity.

Abstract There is a growing demand for correct parallel programs, mainly due to nowadays availability of parallel architectures. Structural testing allows identifying defects by analyzing the internal structures of a program. However, communication and synchronization in parallel programs bring new challenges to the testing activity, such as nondeterminism. Message‐passing parallel programs require structural testing criteria to support test models and tools capable of covering synchronization events with dynamic behaviors. Testing such primitives inside loops in message‐passing programs is still challenging with nontrivial solutions for criteria, models, and tools. This article proposes new structural testing criteria to guide the selection of test cases, improving the quality of message‐passing programs by revealing nondeterminism‐related defects present in loops. We present a new test model to support our criteria for structural testing of MPI‐applications and implement the proposed criteria in the tool ValiMPI. The analysis of nondeterminism‐related defects, paths inside loops, loop iterations, and nested loops allow us to establish a structured testing model. We validate the testing criteria through experimental studies using ValiMPI. Our results show that our criteria reveal unknown defects from communication and synchronization events in different loop iterations, increasing the quality of message‐passing parallel programs.

Mutation testing is well known as one of the most effective approaches to create test cases, which can detect software faults. However, its drawback is the low scalability — if no special attention is given to improve efficiency — that directly affects its application in practice. This paper shows a replication study focused on emphasizing evidence in which the use of distributed processing structures can improve mutation testing. For this purpose, an architecture that enables mutation testing concurrent execution was designed. Five load balancing algorithms responsible for controlling the distribution and execution of data while carrying out mutation testing were evaluated. Experiments were conducted in order to evaluate the scalability and performance of the architecture considering homogeneous and heterogeneous setups. A time reduction of 50% was observed when executing mutants in parallel in relation to the conventional sequential application of mutation testing. The performance gain was above 95% when there was a higher number of nodes in the distributed architecture.

Context: Mobile devices have become increasingly popular, and mobile applications should guarantee a very high level of reliability and quality. Mobile application testing needs to consider several unique requirements that distinguish it from conventional software testing. Objective: Our study aims to establish an overview of the testing practices conducted in mobile companies, to identify weaknesses that can be improved to make the testing activity more effective. Method: The survey questions were carefully designed using the Goal/Question/Metric method to provide relevant information to the questions raised in our study. Results and Conclusions: Our study outlines that native applications are more common. The testing level more performed is the system test and the positions that perform testing levels and objectives are described. Practices related to testing technique selection in the context of mobile applications are highlighted. In the context of this study, Cucumber and Selenium are the testing tools most used to automate testing activity. Some mobile testing characteristics were outlined to understand how the testing in mobile applications run on different devices, how testers deal with the diversity of operating systems that are constantly updated and whether tests are unified to testing a mobile app that runs in different platforms. Furthermore, we report the main challenges faced by testers during the validation of the mobile app.

Conversational agents have been established to support educational practices, solving students' questions, recommending teaching materials, increasing the motivation of students who have little interest in certain content, among others. Although many studies are discussing the establishing of conversational agents, there is no global understanding of what has been investigated in this area. There are some previous secondary studies that have attempted to map the conversational agents used for educational purposes (i.e., pedagogical conversational agents). However, these studies are limited to include primary studies published in specific sets of conferences and journals. Therefore, an overview of the current state of the art associated with pedagogical conversational agents has not been produced yet. This research paper aims to contribute to the theme, through the presentation of planning, conduction, and results analysis of a systematic mapping on pedagogical conversational agents. We address the following questions: (i) in which areas have conversation agents been investigated?; (ii) at which education levels are conversational agents used?; (iii) which mechanisms do conversational agents use to interact with students?. We also present some discussions about research opportunities that can be explored in future works. This study aims to contribute to the topic of pedagogical conversational agents, helping researchers to have a deeper understanding of what has already been done and to guide their research to subjects not explored yet.

Multi-tenancy is the key feature for every Software as a Service (SaaS), as it enables multiple customers, so-called tenants, to transparently share a system's resources reducing costs. Tenants can customize a system according to their particular needs, however, such a high level of complexity may open possibilities for a failure. In addition, there is a lack of a reference architecture for such applications and once the implementations differ significantly, ensuring that all executions flows have been verified without impacting the working features for other tenants is a complex task. The clear understanding of the possible faults is fundamental for the identification, tolerance and definition of appropriate testing techniques. This paper presents a preliminary fault taxonomy for multi-tenant cloud applications considering their foundational features. A literature review previously carried out, a survey with practitioners and analysis of some applications were performed to achieve this classification. In addition, an e-commerce called MtShop was developed for a case study. The expressiveness of the proposed taxonomy is illustrated with critical faults identified in the MtShop through the automated and parallel testing. We conclude with the benefits that our taxonomy can bring to testing, prediction and regression testing activity of multi-tenant cloud applications.

Virtualization is an expanding technology in all areas of research and development. It has been widely accepted because of characteristics such as elasticity and flexibility in delivering on-demand resources. Nowadays, containers are a new form of virtualization and they have been highlighted as large technology companies are giving more support to them. Containers are a type of OS-level virtualization, in which the kernel allows the existence of multiple isolated instances. This paper presents a survey of recent research and challenges involving containers, focusing on performance evaluation. To achieve this goal, we performed a systematic mapping considering the main databases (Springer, IEEE, ACM, and Scopus), raising the challenges of the area and prospects of future work, considering the recent use of containers. We have observed that there is still little research related to performance evaluation of containers and we consider this topic very important since performance is fundamental to the design and adoption of projects.

Software Engineering (SE) is an important topic to be taught in Computer Science courses. However, teaching of theoretical concepts with no link to their practical applications or no examples in the student's context may discourage learning, justifying why teaching and learning are great challenges of education in universities. In attempt to bridge such gap, several approaches have been proposed and applied to improve teaching and learning SE such as project based learning (PBL), a well-known approach already applied to teach SE. Nevertheless, there's a lack of understanding about how to better apply PBL and how to take advantage of this approach, for future use. There is also a lack of experience report describing how to use, its challenges and difficulties, what could be hampering to widely adopt it. We present our experience applying a PBL approach combined with project management to create an environment considering aspects such as dealing with managers and real stakeholders. The goal is to bring students closer to the reality of developing a software project in the business context. Our experience indicates positive results on the adoption of a PBL approach. In general, students were enthusiastic and positive about the use of this approach, the presence of a manager and the importance of using real-world problems with real stakeholders.

Search Based Software Testing refers to the use of meta-heuristics for the optimization of a task in the context of software testing. Meta-heuristics can solve complex problems in which an optimum solution must be found among a large amount of possibilities. The use of meta-heuristics in testing activities is promising because of the high number of inputs that should be tested. Previous studies on search based software testing have focused on the application of meta-heuristics for the optimization of structural and functional criteria. Recently, some researchers have proposed the use of SBST for mutation testing and explored solutions for the cost of application of this testing criterion. The objective is to identify how SBST has been explored in the context of mutation testing, how fitness functions are defined and the challenges and research opportunities in the application of meta-heuristic search techniques. A systematic review involving 263 papers published between 1996 and 2014 examined the studies on the use of meta-heuristic search techniques for the optimization of mutation testing. The results show meta-heuristic search techniques have been applied for the optimization of test data generation, mutant generation and selection of effective mutation operators. Five meta-heuristic techniques, namely Genetic Algorithm, Ant Colony, Bacteriological Algorithm, Hill Climbing and Simulated Annealing have been used in search based mutation testing. The review addressed different fitness functions used to guide the search. Search based mutation testing is a field of interest, however, some issues remain unexplored. For instance, the use of meta-heuristics for the selection of effective mutation operators was identified in only one study. The results have pointed a range of possibilities for new studies to be developed, i.e., identification of equivalent mutants, experimental studies and application to different domains, such as concurrent programs.

Background: SaaS (Software as a Service) is a services delivery model in Cloud Computing whose applications are remotely hosted by the service provider and available to customers on demand over the Internet.Multi-tenant Architecture (MTA) is an organizational pattern for SaaS that enables a single instance of an application to be hosted on the same hardware and accessed by multiple customers, so-called tenants, with the aim of lowering costs.Tenants are able to configure the system according to their particular needs.Objective: This research aims at the obtaining an overview of the challenges and research opportunities in MTA context for SaaS through a Systematic Mapping Study.Results: Eighty nine primary studies were selected for discussions on advances and opportunities for further investigations.The results showed the relevancy of MTA and pointed out the main research trends for next years in this topic.

Several methods have been proposed for generating tests from a Finite State Machine (FSM) representation of systems. Even though these methods usually generate large test suites, each with many sequences, the methods possess the important feature of generating test suites with guaranteed fault coverage. In this paper, we propose approaches for reducing the size of a test suite while maintaining the same fault coverage. The approaches combine the test suite sequences, aiming at both reducing the test suite length and the number of sequences, and then check whether the resulting test suite satisfies conditions that are known to ensure that the coverage remains unaltered. The experimental results in random FSMs show on average 78% (maximum: 97.0%) of reduction of the number of test cases and on average 42% (maximum: 63.8%) of reduction of total length of test suites produced by classical FSM-based generation methods.

This paper proposes novel techniques for the extraction of structural information from the source code of Java concurrent programs for their coverage testing. Such techniques differ from others because they consider synchronization flow among processes/threads, distinct paradigms of communication/synchronization, method calls and pointer manipulation. The structural information gathered from the source code is kept in a test model based on a Parallel Control Flow Graph (PCFG) and helps the generation of an instrumented source code, used for a future generation of trace files and to replay the concurrent execution. The results show the techniques can generate both an instrumented code and a PCFG for Java concurrent programs effectively, extracting static and runtime information required for structural testing.

Testing activity applied to mobile robotic systems is a challenge because new features, such as non-determinism of inputs, communication among components and time constraints must be considered.Simulation has been used to support the development and validation of these systems.Coverage testing criteria can contribute to this scenario adding mechanisms for measuring quality during the development of systems.This paper presents a test model and a set of coverage criteria to test the interaction among the components of mobile robotic systems.The model and criteria focus on robotic systems developed in ROS, a Robotic Operational System in which communication is established through publish/subscribe interaction schema.The testing criteria were evaluated using a robotic application.The results confirm that the use of coverage testing criteria has advantages for integration testing of mobile robotic systems.

Summary Testing is a key activity to assure the quality of concurrent applications. In recent years, a variety of different mechanisms have been proposed to test concurrent software. However, a persistent problem is the high testing cost because of the large number of different synchronization sequences that must be tested. When structural testing criteria are adopted, a large number of infeasible synchronization sequences is generated, increasing the testing cost. Although the use of reachability testing reduces the number of infeasible combination (because only feasible synchronization sequences are generated), many synchronization combinations are also generated, and this again results in a testing cost with exponential behavior. This paper presents a new composite approach that uses reachability testing to guide the selection of the synchronization sequences tests according to a specific structural testing criterion. This new composite approach is empirically evaluated in the context of message‐passing concurrent programs developed with MPI. The experimental study evaluates both the cost and effectiveness of proposed composite approach in comparison with traditional reachability testing and structural testing. The results confirm that the use of the new composite approach has advantages for testing of concurrent applications. Copyright © 2015 John Wiley & Sons, Ltd.

Both software testing and programming foundations are difficult subjects to be taught. In case of programming, experiences have suggested that the mastering of testing can contribute to enhance the students' skills of comprehension and analysis. So, automatic test-based assessment systems have been developed and adopted in programming courses. Aiming at investigating the adoption of assessment systems also in the teaching of software testing, in this paper we discuss the use of ProgTest - a tool for the submission and automatic evaluation of programming assignments based on testing activities. As most of the automatic assessment systems, ProgTest offers benefits such as immediate feedback, objectivity and consistency of the evaluation, and a substantial time saving in the evaluation of the programming assignments. Additionally, since it performs the assessments based on testing activities, ProgTest also offers benefits for the teaching of software testing. Results from two experiments involving the use of ProgTest in testing courses are presented, providing evidences on the feasibility of its adoption to support the teaching of testing concepts and criteria.

SUMMARY Developing high‐quality, error‐free message‐passing concurrent programs is not trivial. Although a number of different primitives with associated semantics are available to assist such development, they often increase the complexity of the testing process. In this paper, we extend our previous test model for message‐passing programs and present new structural testing criteria, taking into account additional features used in this paradigm, such as collective communication, non‐blocking sends, distinct semantics for non‐blocking receives, and persistent operations. Our new model also recognizes that sender primitives cannot always be matched with every receive primitive. This improvement allows us to remove statically a significant number of infeasible synchronization edges that would otherwise have to be analyzed later by the tester. In this paper, the test model is presented using the Message‐Passing Interface standard; however, our new model has been designed to be flexible, and it can be configured to support a range of different message‐passing environments or languages. We have carried out case studies showing the applicability of the new test model to represent message‐passing programs and also to reveal errors, mainly those errors related to inter‐process communication. In addition to increasing the number of features supported by the test model, we have also reduced the overall cost of testing significantly. Our case studies suggest that the number of synchronization edges can be reduced by up to 93%, mainly by eliminating infeasible edges between unmatchable communication primitives. The main contribution of the paper is to present a more flexible test model that provides improved coverage for message‐passing programs and at the same time reduces the cost of testing significantly. Copyright © 2012 John Wiley & Sons, Ltd.

Educational modules, concise units of study capable of integrating theoretical/practical content and supporting tools, are relevant mechanisms to improve learning processes. In this paper we briefly discuss the establishment of mechanisms to ease the development ofeducational modules - a Standard Process for Developing Educational Modules and an Integrated Modeling Approach for structuring their learning content. The proposed mechanisms have been investigated in the development of the ITonCode module - an educational module for teaching inspection and testing techniques. Aiming at evaluating the module we have replicated an extended version of the Basili & Selby experiment, originally used for comparing V&V techniques, now considering the educational context. The obtained results were mainly analyzed in terms of the student's uniformity in detecting existent faults, giving us very preliminar evidences on the learning effectiveness provided by the module produced.

Abstract Parallel programs present some features such as concurrency, communication and synchronization that make the test a challenging activity. Because of these characteristics, the direct application of traditional testing is not always possible and adequate testing criteria and tools are necessary. In this paper we investigate the challenges of validating message‐passing parallel programs and present a set of specific testing criteria. We introduce a family of structural testing criteria based on a test model. The model captures control and data flow of the message‐passing programs, by considering their sequential and parallel aspects. The criteria provide a coverage measure that can be used for evaluating the progress of the testing activity and also provide guidelines for the generation of test data. We also describe a tool, called ValiPar, which supports the application of the proposed testing criteria. Currently, ValiPar is configured for parallel virtual machine (PVM) and message‐passing interface (MPI). Results of the application of the proposed criteria to MPI programs are also presented and analyzed. Copyright © 2008 John Wiley & Sons, Ltd.

In this paper we propose a parallel approach for the K-meansVector Quantization (VQ) algorithm used in a two-stageHidden Markov Model (HMM)-based system forrecognizing handwritten numeral strings. With thisparallel algorithm, based on the master/slave paradigm,we overcome two drawbacks of the sequential version: a)the time taken to create the codebook; and b) the amountof memory necessary to work with large trainingdatabases. Distributing the training samples over theslaves' local disks reduces the overhead associated withthe communication process. In addition, modelspredicting computation and communication time havebeen developed. These models are useful to predict theoptimal number of slaves taking into account the numberof training samples and codebook size.