Abstract The security of operating system has always been challenged by the hidden operation of rootkits. Based on virtual machine introspection technology, security tools are deployed outside the target virtual machine (TVM) that provides strict isolation between them and enhances the anti-interference of security tools. However, the current methods based on virtualization can only detect the hidden objects but cannot make them visible to TVM that leads to handling failure for host-based security tools. To solve this problem, this paper proposes a hidden object detection and recovery method RecObj based on virtualization technology. RecObj uses multidimensional semantic views cross-comparison to discover the hidden processes and files. By dynamically monitoring the change of logical relationship between loadable kernel modules and the change of their states, the hiding detection of rootkit itself can be realized. For processes and modules hidden by direct kernel object manipulation technology, RecObj uses memory writable mapping to restore hidden objects to be visible objects. Finally, the processing signal is transmitted to the hidden object manager in TVM through xenstore, and the cleanup operation to the hidden object is completed. The feasibility and effectiveness of RecObj is proved through the hiding detection, recovery, and processing experiments.

API call sequence-based approaches are proven to have significant superiority in malware detection but generally overlook or evade two core issues: (i) ignoring parameters and return values that contain more fine-grained security semantic sensitive information (SSSI) and (ii) handling lengthy API call sequences roughly, causing the poor interpretability and incompleteness of program behavior semantics. To effectively overcome these issues, we propose SIa-CBc, a sensitive intent-assisted and crucial behavior-cognized malware detection method leveraging human brain cognitive theory, which consists of two key modules. (i) SIa divides the vast and heterogeneous SSSI space into a few categories, meanwhile representing the sensitive intents to assist API calls. (ii) CBc extracts crucial snippets from lengthy API call sequences via judgment and multi-step reasoning and further obtains their representations. The embedding representations from the previous two modules are concatenated as the input of ten representative baseline networks. Our experimental results indicate that SIa-CBc achieves an enhancement in malware detection accuracy ranging from 14.08% to 28.01%, reduces the average detection time per sample by 0.28 to 16.29 ms, and improves the defense against adversarial sample attacks by 4.86% to 55.04%. Moreover, SIa-CBc demonstrates outstanding performance compared to recent methods, not only limited to detection but also encompassing enhanced resilience to intricate adversarial tactics, thereby ensuring reliable protection without the need for frequent re-training. This underscores the model’s innovative approach in leveraging human brain cognitive theory-based techniques for heightened security efficacy.