This paper presents SWAT - System-Wide Analysis Toolkit. It is based on open source emulation and debugging projects and implements the approaches for non-intrusive system-wide analysis and debugging: lightweight OS-agnostic virtual machine introspection, full system execution replay, non-intrusive debugging with WinDbg, and full system reverse debugging. These features are based on novel non-intrusive introspection and reverse debugging methods. They are useful for stealth debugging and analysis of the platforms with custom kernels. SWAT includes multi-platform emulator QEMU with additional instrumentation and debugging features, GUI for convenient QEMU setup and execution, QEMU plugin for non-intrusive introspection, and modified version of GDB. Our toolkit may be useful for the developers of the virtual platforms, emulators, and firmwares/drivers/operating systems. Virtual machine intospection approach does not require loading any guest agents and source code of the OS. Therefore it may be applied to ROM-based guest systems and enables using of record/replay of the system execution. This paper includes the description of SWAT components, analysis methods, and some SWAT use cases.

Debugging of device drivers' failures is a very tough task because of kernel panics, blue screens of death, hardware volatility, long periods of time required to expose the bug, perturbation of the drivers by the debugger, and non-determinism of multi-threaded environment. This paper shows how reverse debugging reduces the influence of these factors to the process of drivers debugging. We present reverse debugger as a practical tool, which was tested for i386, x86-64, and ARM platforms, for Windows and Linux guest operating systems. We show that our tool incurs very low overhead (about 10%), which allows using it for debugging of the time sensitive applications. The paper also presents the case study which demonstrates reverse debugging of the USB kernel drivers for Linux.