Hypervisor is a VMM (Virtual Machine Monitor) that creates and runs multiple VMs (Virtual Machines) through abstracting resources from a physical machine. Hypercall is a special and crucial call used in virtualized systems as it serves as a main communication channel between VMs and the hypervisor. However, hypercall attacks occur when an attacker manipulates the communication channel, and it could cause abnormal VM status, potentially leading to the abnormal resource allocation of the host OS (Operating System) and crash of VMs. Therefore, the virtualized system should execute abnormal VM status detection to identify potential abnormal behaviors to protect VMs and the host OS; however, existing works are either for reconstructing the hypervisor or hardware isolation, not for the VM status detection for abnormal hypercall.This study develops a hypercall-oriented abnormal VM status detection system called HypercallDetector based on the following three innovations: 1) we implement a hypercall tracing based on eBPF to obtain the hypercall-related running status (including CPU usage, memory usage, network traffic, etc.) of each VM; 2) we implement a window division technology to divide the VM status into multiple status windows of the same size, and appropriate window size with balanced detection precision (95.0%) and latency (within 8.8 ms) obtained by proposing the window regulator; and 3) we implement a CS-H algorithm (Compressing Sensing for Hypercall) to distinguish whether the VM status is abnormal. HypercallDetector shows higher precision and lower latency than its opponent and consumes only 8.6% CPU of single core and 0.3% memory usage when starting 240 VMs.

Control flow is the execution order of individual statements, instructions, or function calls within an imperative program. Malicious operation of control flow (e.g., tampering with normal function addresses) leads to severe consequences such as data leakage and system crash. Control Flow Integrity (CFI) is a defense restricting the execution order of program within Control Flow Graph (CFG). IndexHooks is an existing CFI solution designed against forward function calls tampering (including direct and indirect jump). This solution constructs a read-only linear function table that stores function addresses during compilation. Then, IndexHooks checks the table to make program jump to the correct target address during runtime. However, IndexHooks faces limitations in backtracking CFG construction, which can lead to excessive memory usage; the linear structure of the function table is vulnerable to brute force tampering. Addressing the limitations of IndexHooks, this study develops an obfuscated CFI solution called LeafHooks. LeafHooks is implemented during compilation by the LLVM compiler, which performs static analysis and instrumentation on the LLVM Intermediate Representation (IR) code of a program. We make the following three innovations: 1) we propose a speculation-free identification method for indirect function calls by linear traversing and analyzing codes to obtain legal function information (function address); 2) we save this information into a function table in the form of a Balanced Binary Tree (also known as AVL), enhancing the fuzzification of function addresses to defend against brute force; 3) we design a method to simulate control tamper attacks on ARM64 architecture to verify the ability of LeafHooks to protection. LeafHooks shows less overhead than state-of-the-art solutions and reduces 2.9% and 0.55% overhead on average using UnixBench and Phoronix, respectively.