With several microservice architectures comprising thousands of web services in total, used to serve 630 million customers, companies like Meituan face several challenges in the verification and validation of their software. The use of automated techniques, especially advanced AI-based ones, could bring significant benefits here. EvoMaster is an open-source test case generation tool for web services, that exploits the latest advances in the field of Search-Based Software Testing research. This paper reports on our experience of integrating the EvoMaster tool in the testing processes at Meituan over almost 2 years (i.e., between October 2021 and July 2023). Two user studies were carried out in 2021 (with two industrial APIs) and in 2023 (with three industrial APIs) to evaluate two versions of EvoMaster (i.e., v1.3.0 and v1.6.1), respectively, in tackling the test generation for industrial web services which are parts of a large e-commerce microservice system. The two user studies involve in total 321,131 lines of code from these five APIs and 27 industrial participants at Meituan. Questionnaires and interviews were carried out in both user studies with the engineers and managers at Meituan. The two user studies demonstrate clear advantages of EvoMaster (in terms of code coverage and fault detection) and the urgent need to have such a fuzzer in industrial microservices testing. Given its clear advantages, EvoMaster now has been integrated into the industrial testing pipelines at Meituan. To study how these results could generalize, a follow up user study was done in 2024 (with EvoMaster v2.0.0) with five engineers in the five different companies. Our results show that, besides their clear usefulness, there are still many critical challenges that the research community needs to investigate to improve performance further.

Microservices is now becoming a promising architecture to build large-scale web services in industry. Due to the high complexity of enterprise microservices, industry has an urgent need to have a solution to enable automated testing of such systems. EvoMaster is an open-source fuzzer, equipped with the state-of-the-art techniques for supporting automated system-level testing of Web APIs. It has been assessed as the most performant tool in two recent empirical studies in terms of line coverage and fault detection. In this paper, we carried out an empirical experiment to investigate how to better apply the state-of-the-art academic prototype (i.e., EvoMaster) in industrial context. We extended the tool to handle seeding of existing industrial tests, and mocking of external services with their data handled as part of the input fuzzing. We studied two configurations of EvoMaster, using two time budgets, on 40 enterprise RPC-based APIs (involving 5.6 million lines of code for their core business logic) at Meituan. Results show that, compared to existing practice of manual system-level testing and tests produced by record and replay of online traffic, EvoMaster demonstrates clear additional benefits. EvoMaster with the best configuration is capable of covering up to 32.4% line coverage, covering more than 10% line coverage on 36 out of 40 (90%) case studies, and identifying on average 3520 potential faults in these 40 APIs. In addition, we also identified and discussed important challenges in fuzzing enterprise microservices that must be addressed in the future.

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include, for example, Thrift, gRPC, SOFARPC, and Dubbo. Testing such systems using RPC communications is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in this article we propose the first approach in the literature, together with an open source tool, for fuzzing modern RPC-based APIs. The approach is in the context of white-box testing with search-based techniques. To tackle schema extraction of various RPC frameworks, we formulate a RPC schema specification along with a parser that allows the extraction from source code of any JVM RPC-based APIs. Then, with the extracted schema we employ a search to produce tests by maximizing white-box heuristics and newly defined heuristics specific to the RPC domain. We built our approach as an extension to an open source fuzzer (i.e., EvoMaster ), and the approach has been integrated into a real industrial pipeline that could be applied to a real industrial development process for fuzzing RPC-based APIs. To assess our novel approach, we conducted an empirical study with two artificial and four industrial web services selected by our industrial partner. In addition, to further demonstrate its effectiveness and application in industrial settings, we report results of employing our tool for fuzzing another 50 industrial APIs autonomously conducted by our industrial partner in their testing processes. Results show that our novel approach is capable of enabling automated test case generation for industrial RPC-based APIs (i.e., 2 artificial and 54 industrial). We also compared with a simple gray-box technique and existing manually written tests. Our white-box solution achieves significant improvements on code coverage. Regarding fault detection, by conducting a careful review with our industrial partner of the tests generated by our novel approach in the selected four industrial APIs, a total of 41 real faults were identified, which have now been fixed. Another 8,377 detected faults are currently under investigation.