Context Database evolution involves processes such as the evolution of the schema, the adaptation of the application to the new schema, and migrations of data to the new or modified structures of the schema. Data migration is particularly crucial in databases where data repetition is common such as the NoSQL column family DBMSs. In these systems, data integrity cannot be enforced from the database side, but instead needs to be maintained from the application side. Database evolution is also affected by data repetition and the absence of data integrity enforcement from the database, as any evolution of the schema requires data migrations to maintain data integrity. Objectives Ensure data integrity in NoSQL column family DBMSs during database evolution by providing specific instructions for the execution of the necessary data migrations. Methods We propose MoDEvo, a model-driven engineering approach that provides a data migration model to ensure data integrity for database evolution in column-family DBMSs. This model is then transformed into an executable script that implements the migration procedures. Results We evaluate MoDEvo by executing data migrations in case studies obtained from open-source projects where the schema evolved. In this evaluation we use Apache Cassandra, the most popular column-family DBMS. Through this evaluation, we verify that the scripts generated from the data migration model effectively maintain data integrity within the database. Conclusion MoDEvo aids database evolution in column family DBMSs by avoiding the incurrence in the creation of inconsistencies and can also detect impossible migrations, thereby preventing errors. There is still room for improvement such as extending the supported databases to other paradigms where data repetition is common and addressing the evolution of the client applications alongside schema evolution.

Moving testing to the Cloud overcomes time/resource constraints by leveraging an unlimited and elastic infrastructure, especially for testing levels like End-to-End (E2E) that require a high number of resources and/or execution time. However, it introduces new challenges to those already faced on-premises, like selecting the most suitable Cloud infrastructure and billing scheme. We propose the RETORCH* test execution model that estimates and compares the monetary cost of executing an E2E test suite with different Cloud alternatives, billing schemes, and test configurations. RETORCH* goes beyond the mere cost billed, and selects the solution that best aligns with the test team strategy using the data of on-premises prior executions and the tester's experience. This cost is broken down into the cost incurred to execute the test suite (testing cost) and possible unused infrastructure (overprovisioning cost). Based on these distinct costs, the test team can compare different Cloud and test configurations. RETORCH* has been evaluated using a real-world application's E2E test suite. We analyze how the different decisions taken when the suite is migrated to the Cloud impact the cost, highlighting how RETORCH* can help the tester during Cloud and test configuration to make a more informed decision.

Among the current technologies to analyse large data, the MapReduce processing model stands out in Big Data. MapReduce is implemented in frameworks such as Hadoop, Spark or Flink that are able to manage the program executions according to the resources available at runtime. The developer should design the program in order to support all possible non-deterministic executions. However, the program may fail due to a design fault. Debugging these kinds of faults is difficult because the data are executed non-deterministically in parallel and the fault is not caused directly by the code, but by its design. This paper presents a framework called MRDebug which includes two debugging techniques focused on the MapReduce design faults. A spectrum-based fault localization technique locates the root cause of these faults analysing several executions of the test case, and a Delta Debugging technique isolates the data relevant to trigger the failure. An empirical evaluation with 13 programs shows that MRDebug is effective in debugging the faults, especially when the localization is done with the reduced data. In summary, MRDebug automatically provides valuable information to understandMapReducedesign faults as it helps locate their root cause and obtains a minimal data that triggers the failure.

Software testing is an essential knowledge area required by industry for software engineers. However, software engineering students often consider testing less appealing than designing or coding. Consequently, it is difficult to engage students to create effective tests. To encourage students, we explored the use of gamification and investigated whether this technique can help to improve the engagement and performance of software testing students. We conducted a controlled experiment to compare the engagement and performance of two groups of students that took an undergraduate software testing course in different academic years. The experimental group is formed by 135 students from the gamified course whereas the control group is formed by 100 students from the non-gamified course. The data collected were statistically analyzed to answer the research questions of this study. The results show that the students that participated in the gamification experience were more engaged and achieved a better performance. As an additional finding, the analysis of the results reveals that a key aspect to succeed is the gamification experience design. It is important to distribute the motivating stimulus provided by the gamification throughout the whole experience to engage students until the end. Given these results, we plan to readjust the gamification experience design to increase student engagement in the last stage of the experience, as well as to conduct a longitudinal study to evaluate the effects of gamification.

In recent years, software applications have been working with NoSQL databases as they have emerged to handle big data more efficiently than traditional databases. The data models of these databases are designed to satisfy the requirements of the software application, which means that the models must evolve when the requirements of the software application change. To avoid mistakes during the design and evolution of these NoSQL models, there are several methodologies that recommend using a conceptual model. This implies that consistency between the conceptual model and the schema must be maintained when either evolving the database or the software application. In this work, we propose CoDEvo, a model-driven engineering approach that uses model transformations to address the evolution of a NoSQL column family DBMS schema when the underlying conceptual model evolves due to software requirement changes, aiming to maintain consistency between the schema and conceptual model. We have addressed this problem by defining transformation rules that determine how to evolve the schema for a specific conceptual model change. To validate these transformations, we applied them to conceptual model changes from 9 open-source software applications, comparing the output schemas from CoDEvo with the schemas that were defined in these applications.

Software testing is an integral and important part of the software engineering discipline, and its automation has been actively pursued in both academia and industry to reduce its high costs. In the past decades, a considerable research effort has been devoted to automatic test case generation, automatic test selection, and automatic test oracles. The practice of software test automation has also moved forward significantly, and in recent years, a large number of software test tools have been developed and released to the market. However, progress in automatic software testing (AST) research is still required. The increasing complexity, pervasiveness and inter-connection of software systems, the ever-shrinking development cycles and time-to-market, and the scarcity of tools that can support all testing tasks within one environment, have posed new challenges and stricter constraints. Thus, despite significant achievements both in theory and practice, AST remains a challenging research area, and there is an urgent requirement to improve test automation to scale up productivity and quality in software development. Many times, however, industry needs differ from the research agenda, as companies need to prioritize reducing cost and time-to-market. Moreover, practitioners may have a hard time choosing a particular testing method or technology, since the real challenges that influence the decision are usually hidden.\nThis special issue includes revised and extended versions of the best papers presented at the 2nd ACM/IEEE International Conference on Automation of Software Test (AST 2021), held in conjunction with the 43rd International Conference on Software Engineering (ICSE 2021), as well as new original submissions on the theme of “Automatic Software Testing from the Trenches.” This issue initially received a total of 13 submissions.\nBoth the extended papers from AST 2021 as well as the new original submissions underwent a rigorous review process, and ultimately, 8 submissions were accepted for inclusion in this special issue.

Database schemas evolve over time to satisfy changing application requirements. If this evolution is not performed correctly, some quality attributes are at risk such as data integrity, functional correctness, or maintainability. To help developer teams in the design of database schemas, several design methodologies for NoSQL databases have proposed to use conceptual models during this process. The use of an explicit conceptual model can also help developers in the tasks of schema evolution. In this work-in-progress paper, we propose a framework that, given a change in the conceptual model, identifies what must be modified in a NoSQL database schema and the underlying data. We researched several open source projects that use Apache Cassandra to study the benefits of using a conceptual model during the schema evolution process as well as to understand how these models evolve. In this first work, we have focused on studying seven types of conceptual model changes identified in these projects. For each change we describe the transformation required in the database schema to maintain the consistency between the schema and the model as well as the migration of data required to the new schema version.

Web application testing is a great challenge due to the management of complex asynchronous communications, the concurrency between the clients-servers, and the heterogeneity of resources employed. It is difficult to ensure that a test case is re-running in the same conditions because it can be executed in undesirable ways according to several environmental factors that are not easy to fine-grain control such as network bottlenecks, memory issues or screen resolution. These environmental factors can cause flakiness, which occurs when the same test case sometimes obtains one test outcome and other times another outcome in the same application due to the execution of environmental factors. The tester usually stops relying on flaky test cases because their outcome varies during the re-executions. To fix and reduce the flakiness it is very important to locate and understand which environmental factors cause the flakiness. This paper is focused on the localization of the root cause of flakiness in web applications based on the characterization of the different environmental factors that are not controlled during testing. The root cause of flakiness is located by means of spectrum-based localization techniques that analyse the test execution under different combinations of the environmental factors that can trigger the flakiness. This technique is evaluated with an educational web platform called FullTeaching. As a result, our technique was able to locate automatically the root cause of flakiness and provide enough information to both understand it and fix it.

International Conference on Web Information Systems and Technologies, WEBIST (15th. 2019. Vienna, Austria)

The use of NoSQL databases for cloud environments has been increasing due to their performance advantages when working with big data.One of the most popular NoSQL databases used for cloud services is Cassandra, in which each table is created to satisfy one query.This means that as the same data could be retrieved by several queries, these data may be repeated in several different tables.The integrity of these data must be maintained in the application that works with the database, instead of in the database itself as in relational databases.In this paper, we propose a method to ensure the data integrity when there is a modification of data by using a conceptual model that is directly connected to the logical model that represents the Cassandra tables.This method identifies which tables are affected by the modification of the data and also proposes how the data integrity of the database may be ensured.We detail the process of this method along with two examples where we apply it in two insertions of tuples in a conceptual model.We also apply this method to a case study where we insert

Summary Context MapReduce is a processing model used in Big Data to facilitate the analysis of large data under a distributed architecture. Objective The aim of this study is to identify and categorize the state of the art of software testing in MapReduce applications, determining trends and gaps. Method Systematic mapping study to discuss and classify according to international standards 54 relevant studies in relation to reasons for testing, types of testing, quality characteristics, test activities, tools, roles, processes, test levels, and research validations. Results The principal reasons for testing MapReduce applications are performance issues, potential failures, issues related to the data, or to satisfy the agreements with efficient resources. The efforts are focused on performance and, to a lesser degree, on functionality. Performance testing is carried out through simulation and evaluation, whereas functional testing considers some program characteristics (such as specification and structure). Despite the type of testing, the majority of efforts are focused at the unit and integration test levels of the specific MapReduce functions without considering other parts of the technology stack. Conclusions Researchers have both opportunities and challenges in performance and functional testing, and there is room to improve their research though the use of mature and standard validation methods.

The use of NoSQL databases has recently been increasing, being Cassandra one of the most popular ones.In Cassandra, each table is created to satisfy one query, so, as the same information could be retrieved by several queries, this information may be found in several distinct tables.However, the lack of mechanisms to ensure the integrity of the data means that the integrity could be broken after a modification of data.In this paper, we propose a method for keeping the integrity of the data by using a conceptual model that is directly connected to the logical model that represents the Cassandra tables.Our proposal aims to keep the data integrity automatically by providing a process that will undertake such maintenance when there is a modification of the data in the database.The conceptual model will be used to identify the tables that could have inconsistencies and also assist in resolving them.We also apply this approach to a case study where, given several insertions of tuples in the conceptual model, we determine what is needed to keep the logical integrity.

Entity reconciliation (ER) aims to combine data from different sources for a unified vision. The management of large volumes of data has given rise to significant challenges to the ER problem due to facts such as data becoming more unstructured, unclean, and incomplete or the existence of many datasets that store information about the same topic. Testing the applications that implement the ER problem is crucial to ensure both the correctness of the reconciliation process and the quality of the reconciled data. This paper presents an approach based on model-driven engineering that allows the creation of test models for the early integration testing of ER applications, contributing in three main aspects: the description of the elements of the proposed framework, the definition of the testing model, and the validation of the proposal through two real-world case studies. This validation verifies that the early integration testing of the ER application is capable of detecting a series of deficiencies, which a priori are not known and that will help to improve the final result that the ER application offers.

New processing models are being adopted in Big Data engineering to overcome the limitations of traditional technology. Among them, MapReduce stands out by allowing for the processing of large volumes of data over a distributed infrastructure that can change during runtime. The developer only designs the functionality of the program and its execution is managed by a distributed system. As a consequence, a program can behave differently at each execution because it is automatically adapted to the resources available at each moment. Therefore, when the program has a design fault, this could be revealed in some executions and masked in others. However, during testing, these faults are usually masked because the test infrastructure is stable, and they are only revealed in production because the environment is more aggressive with infrastructure failures, among other reasons. This paper proposes new testing techniques that aimed to detect these design faults by simulating different infrastructure configurations. The testing techniques generate a representative set of infrastructure configurations that as whole are more likely to reveal failures using random testing, and partition testing together with combinatorial testing. The techniques are automated by using a test execution engine called MRTest that is able to detect these faults using only the test input data, regardless of the expected output. Our empirical evaluation shows that MRTest can automatically detect these design faults within a reasonable time.

Big Data programs are those that process large data exceeding the capabilities of traditional technologies. Among newly proposed processing models, MapReduce stands out as it allows the analysis of schema-less data in large distributed environments with frequent infrastructure failures. Functional faults in MapReduce are hard to detect in a testing/preproduction environment due to its distributed characteristics. We propose an automatic test framework implementing a novel testing approach called Ex Vivo. The framework employs data from production but executes the tests in a laboratory to avoid side-effects on the application. Faults are detected automatically without human intervention by checking if the same data would generate different outputs with different infrastructure configurations. The framework (MrExist) is validated with a real-world program. MrExist can identify a fault in a few seconds, then the program can be stopped, not only avoiding an incorrect output, but also saving money, time and energy of production resources.

NoSQL databases provide high availability and efficiency in data processing but at the expense of weaker consistency. In this paper, we propose a new approach in order to test NoSQL key/value databases in general and their CRUD operations in particular. We design a new context-aware model that takes into account the contextual requirements of clients (users) and the NoSQL database system. Accordingly, we develop a transaction model and testing criteria in order to test NoSQL databases by taking into account transactional and non-transactional CRUD operations. Results from testing criteria are used to analyse the trade-off between availability and consistency of NoSQL databases. In addition, these are used to help NoSQL database users and developers to choose between transactional and non-transactional CRUD operations.

Functional testing of applications that process the information stored in databases often requires a careful design of the test database. The larger the test database, the more difficult it is to develop and maintain tests as well as to load and reset the test data. This paper presents an approach to reduce a database with respect to a set of SQL queries and a coverage criterion. The reduction procedures search the rows in the initial database that contribute to the coverage in order to find a representative subset that satisfies the same coverage as the initial database. The approach is automated and efficiently executed against large databases and complex queries. The evaluation is carried out over two real life applications and a well-known database benchmark. The results show a very large degree of reduction as well as scalability in relation to the size of the initial database and the time needed to perform the reduction.

Summary Web services transactions are used to build efficient and reliable Web applications that are distributed across the Internet and are accessed by multiple simultaneous users. Current research develops various models and protocols to improve the performance and reliability of Web services transactions. However, there is little research on testing the different models and protocols of Web services transactions. This paper presents an abstract transaction model that patterns different Web services transactions standards. This model is capable of deriving concrete models to automatically generate test cases for different Web services transactions standards. The proposed model is implemented as a prototype system and is evaluated using a case of the Jboss Transaction. The evaluation shows that the proposed system has the capability to automatically generate test cases and detect possible failures of transactions running under different Web services transactions standards. Copyright © 2012 John Wiley & Sons, Ltd.

Service level agreements (SLAs) are typically used to specify rules regarding the consumption of services that are agreed between the providers of the service-based applications (SBAs) and their consumers. An SLA includes a list of terms that contain the guarantees that must be fulfilled during the provisioning and consumption of the services. Since the violation of such guarantees may lead to the application of potential penalties, it is important to assure that the SBA behaves as expected. In this paper, we propose a proactive approach to test SLA-aware SBAs by means of identifying test requirements, which represent situations that are relevant to be tested. To address this issue, we define a four-valued logic that allows evaluating both the individual guarantee terms and their logical relationships. Grounded in this logic, we devise a test criterion based on the modified condition decision coverage (MCDC) in order to obtain a cost-effective set of test requirements from the structure of the SLA. Furthermore by analyzing the syntax and semantics of the agreement, we define specific rules to avoid non-feasible test requirements. The whole approach has been automated and applied over an eHealth case study.

Testing a database application is a challenging process where both the database and the user interaction have to be considered in the design of test cases. This paper describes a specification-based approach to guide the design of test inputs (both the test database and the user inputs) for a database application and to automatically evaluate the test adequacy. First, the system specification of the application is modelled: (1) the structure of the database and the user interface are represented in a single model, called Integrated Data Model (IDM), (2) the functional requirements are expressed as a set of business rules, written in terms of the IDM. Then, a MCDC-based criterion is applied over the business rules to automatically derive the situations of interest to be tested (test requirements), which guide the design of the test inputs. Finally, the adequacy of these test inputs is automatically evaluated to determine whether the test requirements are covered. The approach has been applied to the TPC-C benchmark. The results show that it allows designing test cases that are able to detect interesting faults which were located in the procedural code of the implementation.

Service Level Agreements (SLAs) are used to specify the negotiated conditions between the provider and the consumer of services. In this paper we present a stepwise method to identify and categorize a set of test requirements that represent the potential situations that can be exercised regarding the specification of each isolated guarantee term of an SLA. This identification is addressed by means of devising a set of coverage levels that allow grading the thoroughness of the tests. The utilization of these test requirements would focus on twofold objectives: (1) the generation of a test suite that allows exercising the situations described in the test requirements and (2) the support for the derivation of a monitoring plan that checks the compliance of these requirements at runtime. The approach is illustrated over an eHealth case study.

Transactions are a key issue in the reliability of distributed applications because they ensure all the participants achieve a mutually agreed outcome. However, current research has given little attention to testing transactions in web services. This paper presents a conceptual framework, inspired in risk-based methodologies, to address this gap. It also reports on preliminary results and identifies future work.

Transactions are a fundamental technology for building efficient and reliable web service based applications. Various models and protocols have been developed by academic and industrial research community in order to effectively manage web services transactions. We propose a novel abstract model for dynamically modeling distinct web services transaction protocols. Model-based testing techniques can be used on the abstract model in order to automatically generate test scenarios.

Service Oriented Architectures (SOA) have emerged as a promising solution to develop interoperable and highly dynamic applications. In the domain of SOA, Service Level Agreements (SLAs) are used to specify the stipulated terms between the service provider and the consumer. Due to the unique features of this paradigm such as SLA management, testing SOA presents new challenges to researchers because traditional testing techniques need to be suitably adapted. It is important for both stakeholders to develop proactive approaches that allow the detection of problems before they may later derive in SLA violations. This paper tackles the problem of testing the terms of the SLA, specified in WS-Agreement language, by introducing a stepwise method based on a well-known partition testing technique. The approach is illustrated over a case study.

Abstract In the field of database applications a considerable part of the business logic is implemented using a semi‐declarative language: the Structured Query Language (SQL). Because of the different semantics of SQL compared with other procedural languages, the conventional coverage criteria for testing are not directly applicable. This paper presents a criterion specifically tailored for SQL queries (SQLFpc). It is based on Masking Modified Condition Decision Coverage (MCDC) or Full Predicate Coverage and takes into account a wide range of the syntax and semantics of SQL, including selection, joining, grouping, aggregations, subqueries, case expressions and null values. The criterion assesses the coverage of the test data in relation to the query that is executed and it is expressed as a set of rules that are automatically generated and efficiently evaluated against a test database. The use of the criterion is illustrated in a case study, which includes complex queries. Copyright © 2010 John Wiley & Sons, Ltd.

The XML language is becoming the preferred means of data interchange and representation in web based applications. Usually, XML data is stored in XML repositories, which can be accessed efficiently using the standard XPath as query language. However, the specific techniques for testing these queries often ignore the functional testing. This work addresses this problem by introducing a technique based on the category-partition method for the systematic design of test input data for an XPath query. The method permits the automatic identification of categories and choices in the XPath and XML Schema implementations and the construction of constraints in order to obtain complete and valid test cases. The technique is illustrated over a practical example.

Many software applications have a component based on database management systems in which information is generally handled through SQL queries embedded in the application code. When automation of software testing is mentioned in the research, this is normally associated with programs written in imperative and structured languages. However, the problem of automated software testing applied to programs that manage databases using SQL is still an open issue. This paper presents a measurement of the coverage of SQL queries and the tool that automates it. We also show how database test data may be revised and changed using this measurement by means of completing or deleting information to achieve the highest possible value of coverage of queries that have access to the database.

The use of techniques for automating the generation of software test cases is very important as it can reduce the time and cost of this process. The latest methods for automatic generation of tests use metaheuristic search techniques, i.e. genetic algorithms and simulated annealing. There is a great deal of research into the use of genetic algorithms to obtain a specific coverage in software testing but there is none using the metaheuristic Tabu search technique. In this paper, we explain how we have created an efficient testing technique that combines Tabu search with Korel chaining approach. Our technique automatically generates test data in order to obtain branch coverage in software testing.