Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 70% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge transfer among requests. However, naively caching and reusing past responses leads to a big quality drop.

Cloud microservices are being scaled up due to the rising demand for new features and the convenience of cloud-native technologies. However, the growing scale of microservices complicates the remote procedure call (RPC) dependency graph, exacerbates the tail-of-scale effect, and makes many of the empirical rules for detecting the root cause of end-to-end performance issues unreliable. Additionally, existing open-source microservice benchmarks are too small to evaluate performance debugging algorithms at a production-scale with hundreds or even thousands of services and RPCs.

The lack of representative, publicly-available cloud services has been a recurring problem in the architecture and systems communities. While open-source benchmarks exist, they do not capture the full complexity of cloud services. Application cloning is a promising way to address this, however, prior work is limited to CPU-/cache-centric, single-node services, operating at user level.

Cloud applications are increasingly shifting from large monolithic services to complex graphs of loosely-coupled microservices. Despite the advantages of modularity and elasticity microservices offer, they also complicate cluster management and performance debugging, as dependencies between tiers introduce backpressure and cascading QoS violations. Prior work on performance debugging for cloud services either relies on empirical techniques, or uses supervised learning to diagnose the root causes of performance issues, which requires significant application instrumentation, and is difficult to deploy in practice.

Cloud services have recently started undergoing a major shift from monolithic applications, to graphs of hundreds or thousands of loosely-coupled microservices. Microservices fundamentally change a lot of assumptions current cloud systems are designed with, and present both opportunities and challenges when optimizing for quality of service (QoS) and cloud utilization.

Performance unpredictability is a major roadblock towards cloud adoption, and has performance, cost, and revenue ramifications. Predictable performance is even more critical as cloud services transition from monolithic designs to microservices. Detecting QoS violations after they occur in systems with microservices results in long recovery times, as hotspots propagate and amplify across dependent services. We present Seer, an online cloud performance debugging system that leverages deep learning and the massive amount of tracing data cloud systems collect to learn spatial and temporal patterns that translate to QoS violations. Seer combines lightweight distributed RPC-level tracing, with detailed low-level hardware monitoring to signal an upcoming QoS violation, and diagnose the source of unpredictable performance. Once an imminent QoS violation is detected, Seer notifies the cluster manager to take action to avoid performance degradation altogether. We evaluate Seer both in local clusters, and in large-scale deployments of end-to-end applications built with microservices with hundreds of users. We show that Seer correctly anticipates QoS violations 91% of the time, and avoids the QoS violation to begin with in 84% of cases. Finally, we show that Seer can identify application-level design bugs, and provide insights on how to better architect microservices to achieve predictable performance.

Current cloud services are moving away from monolithic designs and towards graphs of many loosely-coupled, single-concerned microservices. Microservices have several advantages, including speeding up development and deployment, allowing specialization of the software infrastructure, and helping with debugging and error isolation. At the same time they introduce several hardware and software challenges. Given that most of the performance and efficiency implications of microservices happen at scales larger than what is available outside production deployments, studying such effects requires designing the right simulation infrastructures. We present j);qSim, a scalable and validated queueing network simulator designed specifically for interactive microser-vices. μqSim provides detailed intra- and inter-microservice models that allow it to faithfully reproduce the behavior of complex, many-tier applications. μqSim is also modular, allowing reuse of individual models across microservices and end-to-end applications. We have validated μqSim both against simple and more complex microservices graphs, and have shown that it accurately captures performance in terms of throughput and tail latency. Finally, we use μqSim to model the tail at scale effects of request fanout, and the performance impact of power management in latency -sensitive microservices.

Collaborative spectrum sensing is an effective method to improve detection rates in cognitive radio networks. However, it is vulnerable to spectrum sensing data falsification (SSDF) attacks when malicious secondary users (SUs) report fraudulent sensing data. In order to improve the robustness, numerous attack prevention schemes have been proposed to identify malicious SUs. Nevertheless, most of them neglect to incentivize SUs to send truthful reports. An incentive method based on peer-prediction is proposed to identify malicious suspects, punish attackers, and incentivize SUs to send truthful reports simultaneously for decision fusion. Moreover, continuous peer-prediction derived from the binary case is introduced, which is capable of preventing attacks in the continuous domain. Theoretical analysis and simulation results demonstrate that honest SUs are rewarded for accurate and truthful sensing results, while malicious SUs incur penalty for making falsified sensing reports. A significant improvement of detection rates is obtained by the proposed scheme when there are no more than half of malicious SUs conducting SSDF attacks.

Collaborative spectrum sensing is an effective method to improve the detection rate in cognitive radio. However, it is vulnerable to spectrum sensing data falsification attacks. In order to improve the robustness, numerous attack prevention schemes have been proposed to identify malicious secondary users (SUs). Nevertheless, most of them neglect to incentivize SUs to send truthful reports. Therefore, an incentive method based on Private-Prior Peer-Prediction with approximate subjective priors is proposed to identify malicious suspects and punish attackers when falsifying the sensing data simultaneously. The theoretical analysis and simulation results demonstrate that honest SUs are rewarded by accurate and truthful sensing results while malicious SUs receive heavy loss for making falsified sensing results. Moreover, a significant improvement of detection rates is demonstrated when there are a large number of malicious SUs conducting cooperative attacks compared to the pure majority rule scheme.