Recent rapid malware growth has exposed the limitations of traditional in-host malware-defense systems and motivated the development of secure virtualization-based out-of-VM solutions. By running vulnerable systems as virtual machines (VMs) and moving security software from inside the VMs to outside, the out-of-VM solutions securely isolate the anti-malware software from the vulnerable system. However, the presence of semantic gap also leads to the compatibility problem in not supporting existing defense software. In this paper, we present process out-grafting, an architectural approach to address both isolation and compatibility challenges in out-of-VM approaches for fine-grained process-level execution monitoring. Specifically, by relocating a suspect process from inside a VM to run side-by-side with the out-of-VM security tool, our technique effectively removes the semantic gap and supports existing user-mode process monitoring tools without any modification. Moreover, by forwarding the system calls back to the VM, we can smoothly continue the execution of the out-grafted process without weakening the isolation of the monitoring tool. We have developed a KVM-based prototype and used it to natively support a number of existing tools without any modification. The evaluation results including measurement with benchmark programs show it is effective and practical with a small performance overhead.

Protecting kernel control data (e.g., function pointers and return addresses) has been a serious issue plaguing rootkit defenders. In particular, rootkit authors only need to compromise one piece of control data to launch their attacks, while defenders need to protect thousands of such values widely scattered across kernel memory space. Worse, some of this data (e.g., return addresses) is volatile and can be dynamically generated at run time. Existing solutions, however, offer either incomplete protection or excessive performance overhead. To overcome these limitations, we present indexed hooks, a scheme that greatly facilitates kernel control-flow enforcement by thoroughly transforming and restricting kernel control data to take only legal jump targets (allowed by the kernel's control-flow graph). By doing so, we can severely limit the attackers' possibility of exploiting them as an infection vector to launch rootkit attacks. To validate our approach, we have developed a compiler-based prototype that implements this technique in the FreeBSD 8.0 kernel, transforming 49 025 control transfer instructions (~7.25% of the code base) to use indexed hooks instead of direct pointers. Our evaluation results indicate that our approach is generic, effective, and can be implemented on commodity hardware with a low performance overhead (<;5% based on benchmarks).

Virtual machine (VM) introspection is a powerful technique for determining the specific aspects of guest VM execution from outside the VM. Unfortunately, existing introspection solutions share a common questionable assumption. This assumption is embodied in the expectation that original kernel data structures are respected by the untrusted guest and thus can be directly used to bridge the well-known semantic gap. In this paper, we assume the perspective of the attacker, and exploit this questionable assumption to subvert VM introspection. In particular, we present an attack called DKSM (Direct Kernel Structure Manipulation), and show that it can effectively foil existing VM introspection solutions into providing false information. By assuming this perspective, we hope to better understand the challenges and opportunities for the development of future reliable VM introspection solutions that are not vulnerable to the proposed attack.

Multi-field packet classification is frequently performed by network devices such as edge routers and firewalls - such devices can utilize programmable network processors to perform this computation-intensive task at nearly line speeds. The architectures of programmable network processors are typically highly parallel and a single algorithm can be mapped in different ways onto the hardware. We study the performance of two different design mappings of the bit vector packet classification algorithm on the Intel/sup /spl reg// IXP1200 network processor. We show that: (i) overall, parallel mapping has a better packet processing rate (25% more) than pipelined mapping; (ii) in parallel mapping, a processing element's utilization can be considerably affected by code complexity, in terms of branching, because of significant time wasted (as much as 40% more) due to aborting instruction execution pipelines; (iii) in pipelined mapping, multiple memory reads per packet can lower the overall performance.