Modern systems generate a massive amount of logs to detect and diagnose system faults, which incurs expensive storage costs and runtime overhead. After investigating real-world production logs, we observe that most of the logging overhead is due to a small number of log templates, referred to as log hotspots. Therefore, we conduct a systematical study about log hotspots in an industrial system WeChat, which motivates us to identify log hotspots and reduce them on the fly. In this paper, we propose LogReducer, a non-intrusive and language-independent log reduction framework based on eBPF (Extended Berkeley Packet Filter), consisting of both online and offline processes. After two months of serving the offline process of LogReducer in WeChat, the log storage overhead has dropped from 19.7 PB per day to 12.0 PB (i.e., about a 39.08% decrease). Practical implementation and experimental evaluations in the test environment demonstrate that the online process of LogReducer can control the logging overhead of hotspots while preserving logging effectiveness. Moreover, the log hotspot handling time can be reduced from an average of 9 days in production to 10 minutes in the test with the help of LogReducer,

Modern large-scale service-based systems such as microservice systems have become increasingly complex, making it hard to localize anomalous services when various issues emerge. Traces record the workflows of requests through service instances and have been widely used in anomaly detection and root cause analysis. Existing trace-based approaches widely use statistical methods or learning-based techniques to detect trace anomalies and localize anomalous services. However, these approaches often suffer from the concept drift problem, i.e., the statistical properties of traces change over time in unforeseen ways. In this paper, we propose TraceStream, an anomalous service localization approach based on trace data stream clustering. TraceStream uses data stream clustering to discover potential anomalous trace clusters in evolving trace data and uses spectrum analysis to localize anomalous services based on the clusters. Moreover, TraceStream can effectively incorporate the online feedback of operation engineers based on the trace clusters to improve the accuracy for localizing anomalous services. Our evaluation confirms that TraceStream can effectively detect anomalies and localize anomalous services in an evolving microservice system. It can effectively incorporate human feedback to further improve the performance of anomalous service localization. Moreover, TraceStream is efficient and its efficiency can be further improved by sampling a small portion of traces by cluster.

Large-scale online systems are complex, fast-evolving, and hardly bug-free despite the testing efforts. Backend system monitoring cannot detect many types of issues, such as UI related bugs, bugs with small impact on backend system indicators, or errors from third-party co-operating systems, etc. However, users are good informers of such issues: They will provide their feedback for any types of issues. This experience paper discusses our design of iFeedback, a tool to perform real-time issue detection based on user feedback texts. Unlike traditional approaches that analyze user feedback with computation-intensive natural language processing algorithms, iFeedback is focusing on fast issue detection, which can serve as a system life-condition monitor. In particular, iFeedback extracts word combination-based indicators from feedback texts. This allows iFeedback to perform fast system anomaly detection with sophisticated machine learning algorithms. iFeedback then further summarizes the texts with an aim to effectively present the anomaly to the developers for root cause analysis. We present our representative experiences in successfully applying iFeedback in tens of large-scale production online service systems in ten months.

Mobile applications, or apps for short, are gaining popularity. The input sources (e.g., touchscreen, sensors, transmitters) of the smart devices that host these apps enable the apps to offer a rich experience to the users, but these input sources pose testing complications to the developers (e.g., writing tests to accurately utilize multiple input sources together and be able to replay such tests at a later time). To alleviate these complications, researchers and practitioners in recent years have developed a variety of record-and-replay tools to support the testing expressiveness of smart devices. These tools allow developers to easily record and automate the replay of complicated usage scenarios of their app. Due to Android's large share of the smart-device market, numerous record-and-replay tools have been developed using a variety of techniques to test Android apps. To better understand the strengths and weaknesses of these tools, we present a comparison of popular record-and-replay tools from researchers and practitioners, by applying these tools to test three popular industrial apps downloaded from the Google Play store. Our comparison is based on three main metrics: (1) ability to reproduce common usage scenarios, (2) space overhead of traces created by the tools, and (3) robustness of traces created by the tools (when being replayed on devices with different resolutions). The results from our comparison show which record-and-replay tools may be the best for developers and identify future directions for improving these tools to better address testing complications of smart devices.

In recent years, the Software as a Service, largely enabled by the Internet, has become an innovative software delivery model. During the streaming execution of virtualization software, the execution will wait until the missing data was downloaded, which greatly influences the user experience. In this paper, we present a block-level prefetching framework for streaming delivery of software based on N-Gram prediction model and an incremental data mining algorithm. The prefetching framework uses the historical block access logs for data mining, then dynamically updates and polishes the prefetching rules. The experimental results show that this prefetching framework achieves a launch time reduced by 10% to 50%, as well as hit rate between 81% and 97%.

The deterministic annealing method for training the center vectors of RBF networks is proposed. The method is a soft-competition scheme and derived from optimizing an objective function using the gradient descent method. To some extent it can overcome the problems that the learning vector quantization algorithms with the winner-take-all scheme and the heuristic procedure have. The emulation experiment is given to validate the algorithm. The experimental results show that, compared to the error backpropagating algorithms of the multi-layer perception and the RBF network, it not only enhances learning precision and generalization ability, but also reduces learning time as well.