Modern rootkits leverage kernel privileges to hide from analysis tools, while obfuscation techniques render them resistant to static analysis. Reverse engineering malware requires observing memory usage and reconstructing data structures. Existing tools rely on instrumentation or emulation, which introduce high overhead, leave detectable artifacts, and cannot reliably analyze kernel-level malware. We present The Reversing Machine (TRM), a hypervisor-based framework for high-performance introspection of evasive malware. It is the first system to support selective memory tracing and structure reconstruction in the hypervisor. TRM repurposes hardware virtualization to efficiently detect user-kernel mode transitions and obtain memory traces independently of a potentially compromised guest kernel. TRM introduces new insights into leveraging hardware virtualization for runtime memory reconstruction and analysis of data structures while remaining invisible to malware. We demonstrate automatic reconstruction of function signatures and data structures, reduce system call latency overhead from 142% to 57% compared to prior work, and accelerate manual reverse engineering by 43% on average, even for complex kernel objects. TRM shows that hypervisor-level memory tracing makes data structure reconstruction practical in hostile environments, bridging the gap between prior feasibility studies and real-world malware analysis.

Digital Rights Management (DRM) technologies underpin the protection of modern digital content, including music, films, software, games, and e-books, and support multibillion-dollar industries that rely on its effectiveness. In this paper, we question whether this trust in DRM is warranted. In the absence of malicious content-capturing hardware, it rests on the assumption that DRM forces content pirates to resort to the "Analog Hole", where the conversion from digital to analog and back leads to significant degradation in quality. We show that this assumption is false and that even the most sophisticated designs and hardware-level protection of high-quality audio is fundamentally vulnerable to what we term the software-based "Digital Hole". In particular, our hypervisor-based solution intercepts digital communication between kernel space and hardware-based audio peripherals—well beyond the reach of audio DRM technology. As an example, we investigate HD Audio-compatible devices and demonstrate that it is possible to dump DRM-protected songs and convert them to lossless audio files across major commercially available streaming services (Netflix, Spotify, etc.), along with a proof-of-concept for effectively extracting and storing protected content. We analyze the technical roots of this weakness, discuss potential countermeasures, and highlight the broader implications for designing more resilient audio DRM systems. Our work underscores that audio DRM, in its current form, cannot fully achieve its intended goals, motivating the community to radically redesign approaches to digital content protection.

We propose Nucleus, a novel function detection algorithm for binaries. In contrast to prior work, Nucleus is compiler-agnostic, and does not require any learning phase or signature information. Instead of scanning for signatures, Nucleus detects functions at the Control Flow Graph-level, making it inherently suitable for difficult cases such as non-contiguous or multi-entry functions. We evaluate Nucleus on a diverse set of 476 C and C ++ binaries, compiled with gcc, clang and Visual Studio for x86 and x64, at optimization levels O0-O3. We achieve consistently good performance, with a mean F-score of 0.95.

Many reverse engineering techniques for data structures rely on the knowledge of memory allocation routines. Typically, they interpose on the system’s malloc and free functions, and track each chunk of memory thus allocated as a data structure. However, many performance-critical applications implement their own custom memory allocators. Examples include webservers, database management systems, and compilers like gcc and clang. As a result, current binary analysis techniques for tracking data structures fail on such binaries. We present MemBrush, a new tool to detect memory allocation and deallocation functions in stripped binaries with high accuracy. We evaluated the technique on a large number of real world applications that use custom memory allocators. We demonstrate that MemBrush can detect allocators/deallocators with a high accuracy which is 52 out of 59 for allocators, and 29 out of 31 for deallocators in SPECINT 2006. As we show, we can furnish existing reverse engineering tools with detailed information about the memory management API, and as a result perform an analysis of the actual application specific data structures designed by the programmer. Our system uses dynamic analysis and detects memory allocation and deallocation routines by searching for functions that comply with a set of generic characteristics of allocators and deallocators.

Many existing techniques for reversing data structures in C/C ++ binaries are limited to low-level programming constructs, such as individual variables or structs. Unfortunately, without detailed information about a program's pointer structures, forensics and reverse engineering are exceedingly hard. To fill this gap, we propose MemPick, a tool that detects and classifies high-level data structures used in stripped binaries. By analyzing how links between memory objects evolve throughout the program execution, it distinguishes between many commonly used data structures, such as singly- or doubly-linked lists, many types of trees (e.g., AVL, red-black trees, B-trees), and graphs. We evaluate the technique on 10 real world applications, 4 file system implementations and 16 popular libraries. The results show that MemPick can identify the data structures with high accuracy.

Current Control-Flow Integrity (CFI) implementations track control edges individually, insensitive to the context of preceding edges. Recent work demonstrates that this leaves sufficient leeway for powerful ROP attacks. Context-sensitive CFI, which can provide enhanced security, is widely considered impractical for real-world adoption. Our work shows that Context-sensitive CFI (CCFI) for both the backward and forward edge can be implemented efficiently on commodity hardware. We present PathArmor, a binary-level CCFI implementation which tracks paths to sensitive program states, and defines the set of valid control edges within the state context to yield higher precision than existing CFI implementations. Even with simple context-sensitive policies, PathArmor yields significantly stronger CFI invariants than context-insensitive CFI, with similar performance.

Parallax is a novel self-contained code integrity verification approach, that protects instructions by overlapping Return-Oriented Programming (ROP) gadgets with them. Our technique implicitly verifies integrity by translating selected code (verification code) into ROP code which uses gadgets scattered over the binary. Tampering with the protected instructions destroys the gadgets they contain, so that the verification code fails, thereby preventing the adversary from using the modified binary. Unlike prior solutions, Parallax does not rely on code checksumming, so it is not vulnerable to instruction cache modification attacks which affect checksumming techniques. Further, unlike previous algorithms which withstand such attacks, Parallax does not compute hashes of the execution state, and can thus protect code with non-deterministic state. Parallax limits performance overhead to the verification code, while the protected code executes at its normal speed. This allows us to protect performance-critical code, and confine the slowdown to other code regions. Our experiments show that Parallax can protect up to 90% of code bytes, including most control flow instructions, with a performance overhead of under 4%.

This paper evaluates pointer tainting, an incarnation of Dynamic Information Flow Tracking (DIFT), which has recently become an important technique in system security. Pointer tainting has been used for two main purposes: detection of privacy-breaching malware (e.g., trojan keyloggers obtaining the characters typed by a user), and detection of memory corruption attacks against non-control data (e.g., a buffer overflow that modifies a user's privilege level). In both of these cases the attacker does not modify control data such as stored branch targets, so the control flow of the target program does not change. Phrased differently, in terms of instructions executed, the program behaves 'normally'. As a result, these attacks are exceedingly difficult to detect. Pointer tainting is considered one of the onlymethods for detecting them in unmodified binaries. Unfortunately, almost all of the incarnations of pointer tainting are flawed. In particular, we demonstrate that the application of pointer tainting to the detection of keyloggers and other privacybreaching malware is problematic. We also discuss whether pointer tainting is able to reliably detect memory corruption attacks against non-control data. Pointer tainting generates itself the conditions for false positives. We analyse the problems in detail and investigate various ways to improve the technique. Most have serious drawbacks in that they are either impractical (and incur many false positives still), and/or cripple the technique's ability to detect attacks. In conclusion, we argue that depending on architecture and operating system, pointer tainting may have some value in detecting memory orruption attacks (albeit with false negatives and not on the popular x86 architecture), but it is fundamentally not suitable for automated detecting of privacy-breaching malware such as keyloggers.

Heap and stack buffer overflows are still among the most common attack vectors in intrusion attempts. In this paper, we ask a simple question that is surprisingly difficult to answer: which bytes contributed to the overflow? By careful observation of all scenarios that may occur in overflows, we identified the information that needs to be tracked to pinpoint the offending bytes. There are many reasons why this is a hard problem. For instance, by the time an overflow is detected some of the bytes may already have been overwritten, creating gaps. Additionally, it is hard to tell the offending bytes apart from unrelated network data. In our solution, we tag data from the network with an age stamp whenever it is written to a buffer. Doing so allows us to distinguish between different bytes and ignore gaps, and provide precise analysis of the offending bytes. By tracing these bytes to protocol fields, we obtain accurate signatures that cater to polymorphic attacks.

As modern operating systems and software become larger and more complex, they are more likely to contain bugs, which may allow attackers to gain illegitimate access. A fast and reliable mechanism to discern and generate vaccines for such attacks is vital for the successful protection of networks and systems. In this paper we present Argos, a containment environment for worms as well as human orchestrated attacks. Argos is built upon a fast x86 emulator which tracks network data throughout execution to identify their invalid use as jump targets, function addresses, instructions, etc. Furthermore, system call policies disallow the use of network data as arguments to certain calls. When an attack is detected, we perform 'intelligent' process- or kernel-aware logging of the corresponding emulator state for further offline processing. In addition, our own forensics shellcode is injected, replacing the malevolent shellcode, to gather information about the attacked process. By correlating the data logged by the emulator with the data collected from the network, we are able to generate accurate network intrusion detection signatures for the exploits that are immune to payload mutations. The entire process can be automated and has few if any false positives, thus rapid global scale deployment of the signatures is possible.