The advent of cloud computing and inexpensive multi-core desktop architectures has led to the widespread adoption of virtualization technologies. Furthermore, security researchers embraced virtual machine monitors (VMMs) as a new mechanism to guarantee deep isolation of untrusted software components, which, coupled with their popularity, promoted VMMs as a prime target for exploitation. In this paper, we present HyperCheck, a hardware-assisted tampering detection framework designed to protect the integrity of hypervisors and operating systems. Our approach leverages System Management Mode (SMM), a CPU mode in ×86 architecture, to transparently and securely acquire and transmit the full state of a protected machine to a remote server. We have implement two prototypes based on our framework design: HyperCheck-I and HyperCheck-II, that vary in their security assumptions and OS code dependence. In our experiments, we are able to identify rootkits that target the integrity of both hypervisors and operating systems. We show that HyperCheck can defend against attacks that attempt to evade our system. In terms of performance, we measured that HyperCheck can communicate the entire static code of Xen hypervisor and CPU register states in less than 90 million CPU cycles, or 90 ms on a 1 GHz CPU.

Due to performance constraints, host intrusion detection defenses depend on event and polling-based tamper-proof mechanisms to detect security breaches. These defenses monitor the state of critical software components in an attempt to discover any deviations from a pristine or expected state. The rate and type of checks depend can be both periodic and event-based, for instance triggered by hardware events. In this paper, we demonstrate that all software and hardware-assisted defenses that analyze non-contiguous state to infer intrusions are fundamentally vulnerable to a new class of attacks, we call “evasion attacks”. We detail two categories of evasion attacks: directly-intercepting the defense triggering mechanism and indirectly inferring its periodicity. We show that evasion attacks are applicable to a wide-range of protection mechanisms and we analyze their applicability in recent state-of-the-art hardware-assisted protection mechanisms. Finally, we quantify the performance of implemented proof-of-concept prototypes for all of the attacks and suggest potential countermeasures.