Dynamic performance analysis plays a crucial role in optimizing systems and identifying performance bottlenecks. Traditional software debugging methods frequently encounter difficulties when trying to pinpoint performance problems in complex software settings. This is often because performance issues remain hidden during the code execution within debugging tools or under certain run-time circumstances, making them challenging to identify and address. This paper introduces PASD (Performance Analysis through Statistical Debugging), a dynamic performance analysis approach based on statistical debugging of kernel-level trace events. Importantly, this approach requires no application code instrumentation and purely utilizes operating system kernel trace events for analysis. PASD collects kernel trace events generated during software execution and utilizes heuristics to analyze their performance issues and the root-causes. Through statistical debugging techniques, PASD identifies the most important functions correlated with performance problems. It notably does so without disrupting the software’s normal functions and ensuring that any issues are detected in the software’s typical operating conditions, thus avoiding additional complexity in the debugging process. We have conducted two empirical studies to assess the effectiveness of PASD on performance issues in the Firefox web browser as well as the ‘ls’ tool (a common utility in Unix-like systems). Our experiments demonstrate that PASD successfully identifies performance issues and their causes in software without prior knowledge of the architecture or source code instrumentation. By providing an overview of software behavior through the kernel-level, our proposed method can aid developers and testers in quickly pinpointing performance problems in the source code. This, in turn, can result in improved software quality, increased user satisfaction, and the prevention of critical system failures.

Troubleshooting system performance issues is a challenging task that requires a deep understanding of various factors that may impact system performance. This process involves analyzing trace logs from the kernel and user space using tools such as ftrace, strace, DTrace, or LTTng. However, pre-set tracing instrumentation can lead to missing important data where not enough components of the system include observability coverage. Also, having too much coverage may result in unnecessary noise in the data, making it extremely difficult to debug. This paper proposes an adaptive instrumentation technique for execution tracing, which dynamically makes decisions not only for which components to trace but also when to trace, thus reducing the risk of missing important data related to the performance problem and increasing the accuracy of debugging by reducing unwanted noises. Our case study results show that the proposed method is capable of handling tracing instrumentation dynamically for both kernel and application levels while maintaining a low overhead.