Managing complex cloud services requires standard operational documentation, but its sheer volume often hinders cloud engineers from efficient knowledge acquisition. Retrieval-Augmented Generation (RAG) can streamline this process by retrieving relevant knowledge and generating concise, referenced answers. However, deploying a reliable RAG-based chatbot for cloud operation remains a challenge. In this experience paper, we analyze the development and deployment of RAG-based chatbots for operational question answering (OpsQA) at a large-scale cloud vendor. Through an empirical study of 2,000 real-world queries across three operational teams, we identify five unique OpsQA intent types (e.g., symptom analysis and terminology explanation) and their corresponding requirements for a satisfactory answer, which differ from general software engineering queries. Our analysis further uncovers six root causes leading to chatbot failures—over half stem from query issues (i.e., incompleteness, out-of-scope, or invalid queries), while others are from retrieval or generation issues. To address these issues, we propose iKnow, an intent-guided RAG-based chatbot that integrates intent detection, query rewriting tailored to each intent, and missing knowledge detection to enhance answer quality. In internal evaluations, iKnow improves average answer accuracy from 65.8% to 81.3% with only a modest increase in latency. iKnow has been deployed for six months at CloudA, supporting thousands of cloud engineers in daily operations. We discuss lessons learned from real-world deployment, providing valuable insights for future research and practical implementations in similar domains.

Nowadays, online Internet services are pervasive and can be invoked from diverse locations in anytime with multitudinous devices. Conventional testing approaches for online services like Web services are conducted by professional tester or developers and cannot simulate the real world running environment of a service. Fortunately, crowdtesting technology brings us promising hope and has acquired increasing interests and adoption because it can recruit plenty of end users to test services under real world environment with low cost. Meanwhile, improved mobile network techniques make crowdsourcing happen anywhere and anytime. In this paper, we present iTest which combines mobile crowdsourcing and web service testing together to support the performance testing of web services. iTest is a framework for service developers to submit their web services and conveniently get the test results from the crowd testers. Firstly, we analyze the key problems need to be solved in a mobile crowdtesting platform; secondly, the architecture of iTest framework and the workflow in it are presented; Thirdly, we perform experiments to illustrate that both the way to access network and tester's location influence the performance of web service, and formulate the tester selection problem as a Set Cover Problem and propose a greedy algorithm for solving this problem; Next, experimental evaluation of the tester selection algorithm is performed to illustrate its efficiency. Finally, we conclude our work and provide the directions for future work.

Summary In this paper, we present WS‐TaaS, a Web services load testing platform built on a global platform PlanetLab. WS‐TaaS enables load testing process to be simple, transparent, and as close as possible to the real running scenarios of the target services. First, we briefly introduce the base of WS‐TaaS, Service4All. Second, we provide detailed analysis of the requirements of Web service load testing and present its conceptual architecture as well as algorithm design for improving resource utilization. Third, we present the implementation details of WS‐TaaS. Finally, we perform the evaluation of WS‐TaaS with a set of experiments based on the testing of real Web services, and the results illustrate that WS‐TaaS can efficiently facilitate the whole process of Web service load testing. Especially, comparing with existing testing tools, WS‐TaaS can obtain more effective and accurate test results. Copyright © 2014 John Wiley & Sons, Ltd.

Software developers with service oriented technologies usually put a lot of efforts to deploy and manage supporting middleware and tools. Meanwhile PaaS in cloud computing aims at provide efficient support for software developers. In the light of this consideration, we have designed and implemented Service4All, a service cloud platform targeting at improve productivity of service oriented software developers. In this paper, we describe the design of SAE, a key component in Service4All, in terms of scalability. First, we present the key technical issues and architecture design of SAE. Second, we describe a software appliance based mechanism for elastic middleware management. Third, we describe a micro-kernel based AppEngine core for efficient coordination of various components. Finally, through a real application deployed on Service4All, we demonstrate the effectiveness of our solution.

Web services are widely known as the building blocks of typical service oriented applications. The performance of such an application system is mainly dependent on that of component web services. Thus the effective load testing of web services is of great importance to understand and improve the performance of a service oriented system. However, existing Web Service load testing tools ignore the real characteristics of the practical running environment of a web service, which leads to inaccurate test results. In this work, we present WS-TaaS, a load testing platform for web services, which enables load testing process to be as close as possible to the real running scenarios. In this way, we aim at providing testers with more accurate performance testing results than existing tools. WS-TaaS is developed on the basis of our existing Cloud PaaS platform: Service4All. First, we provide detailed analysis of the requirements of Web Service load testing and present the conceptual architecture and design of key components. Then we present the implementation details of WS-TaaS on the basis of Service4All. Finally, we perform a set of experiments based on the testing of real web services, and the experiments illustrate that WS-TaaS can efficiently facilitate the whole process of Web Service load testing.

Web services are widely known as the building blocks of typical service oriented applications. The performance of such an application system is mainly dependent on that of component web services. Thus the effective load testing of web services is of great importance to understand and improve the performance of a service oriented system. However, existing Web Service load testing tools ignore the real characteristics of the practical running environment of a web service, which leads to inaccurate test results. In this work, we present WS-TaaS, a load testing platform for web services, which enables load testing process to be as close as possible to the real running scenarios. In this way, we aim at providing testers with more accurate performance testing results than existing tools. WS-TaaS is developed on the basis of our existing Cloud PaaS platform: Service4All. First, we briefly introduce the functionalities and main components of Service4All. Second, we provide detailed analysis of the requirements of Web Service load testing and present the conceptual architecture and design of key components. Third, we present the implementation details of WS-TaaS on the basis of Service4All. Finally, we perform a set of experiments based on the testing of real web services, and the experiments illustrate that WS-TaaS can efficiently facilitate the whole process of Web Service load testing. Especially, comparing with existing testing tools, WS-TaaS can obtain more effective and accurate test results.