While several test case selection algorithms (heuristic and optimal) and formulations (linear and non-linear) have been proposed, no multi-criteria framework enables Pareto search — the state-of-the-art approach of doing multi-criteria optimization. Therefore, we introduce the highly parallelizable, openly available Many-Criteria Test-Optimization Algorithm (MC-TOA) framework that combines heuristic Pareto search and optimality gap knowledge per criterion. MC-TOA is largely agnostic to the criteria formulations and can incorporate many criteria where existing approaches offer limited scope (single or few objectives/constraints), lack flexibility in the expression and assurance of constraints, or run into problem complexity issues. For two large-scale systems with up to seven criteria and thousands of system test cases, MC-TOA not only produces, over the board, superior Pareto fronts in terms of HVI score compared to the state-of-the-art many-objective heuristic baseline, it also does that within minutes of runtime for worst-case executions, i.e., assuming that a regression affects the entire test-suite. MC-TOA depends on convex solvers. We find that the evaluated open-source solvers are slower but suffice for smaller systems, while being less robust for larger systems. Linear formulations execute faster and obtain near-optimal results, which led to faster and better overall convergence of MC-TOA compared to integer formulations. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

RESTful APIs are arguably the most popular endpoints for accessing Web services. Blackbox testing is one of the emerging techniques for ensuring the reliability of RESTful APIs. The major challenge in testing RESTful APIs is the need for correct sequences of API operation calls for in-depth testing. To build meaningful operation call sequences, researchers have proposed techniques to learn and utilize the API dependencies based on OpenAPI specifications. However, these techniques either lack the overall awareness of how all the APIs are connected or the flexibility of adaptively fixing the learned knowledge.

Many big companies are providing cloud services through RESTful APIs nowadays. With the growing popularity of RESTful API, testing RESTful API becomes crucial. To address this issue, researchers have proposed several automatic RESTful API testing techniques. At Huawei, we design and implement an automatic RESTful API testing framework named Morest. Morest has been used to test ten RESTful API services and helped to detected 83 previously unknown bugs which were all confirmed and fixed by the developers. On one hand, we find that Morest shows great capability of detecting bugs in RESTful API s. On the other hand, we also notice that human effort is inevitable and important when applying automatic RESTful API techniques in practice.

Implementing test cases as programs to automate test execution is a popular testing practice. Current industrial practices usually use test functions to implement the test steps of a test case and then to compose the executable test case by choosing the test functions to call manually. It is time-consuming and could lead to invalid test results by selecting inappropriate test functions. In this article, we propose an automatic test function recommendation approach named Scenario-based Recommendation of TEst Function (SRTEF). Given a test step of a test case, SRTEF uses the weighted description similarity and the scenario similarity to recommend test functions. The description similarity utilizes the deep structured semantic model (DSSM) to measure the relatedness between a test step and a test function by their literal descriptions. The test scenario and the test function usage scenario are considered to calculate the scenario similarity. SRTEF has been successfully applied in Huawei. The systematic experiments have been conducted to evaluate SRTEF by using the dataset from Huawei and comparing with BiInformation source-based KnowledgE Recommendation (BIKER), reported as the best approach so far. The results show that SRTEF outperforms BIKER with significant positive ratios consistently in all the three selection strategies, i.e., Top-3, Top-5, and Top-10. The DSSM shows its advantage over word embedding by the double performance of capturing the semantic relatedness in SRTEF.

Implementing test cases to automate test execution is a popular testing practice currently. A stepwise test case consists of several sequential test steps. Given a test function library, the typical way to implement a test case is calling the existing test functions in the library to reduce test cost. How to find the appropriate test function(s) to implement a test step in a given test case thus becomes an important problem. However, in current testing practices, test engineers usually select the appropriate test function manually by experience. It is time-consuming and could lead to invalid test results by selecting inappropriate or wrong test functions to call. In this paper, we propose an automatic test function recommendation approach with scenario named SRTEF (Scenario-based Recommendation of TEst Function). Given a test step, SRTEF uses two levels of similarities to recommend test functions, description similarity and scenario similarity. The description similarity measures the semantic relatedness between the test step and test function by their literal descriptions. To calculate the scenario similarity, SRTEF at first retrieves a set of historical test cases that contains test step(s) semantically similar to the given test step; then the scenario similarity between test step and test function is calculated according to the calling relation between retrieved test case and test function, and the co-occurrence relation among test functions. SRTEF has been successfully applied in Huawei. We evaluate SRTEF by using the dataset from Huawei and comparing with BIKER, reported as the best recommendation approach so far. The results show that SRTEF outperforms the BIKER approach by at least 49% in Mean Average Precision, 33% in Mean Reciprocal Rank, and 25% in Mean Recall.