The detection and classification of packers serve as a fundamental approach in the study of malware unpacking. In our research, we employ Hardware Performance Counters (HPCs) as features for our classification process. Hardware Performance Counters are integrated into a processor’s Performance Monitoring Unit. The advantages of utilizing hardware performance counters include low overhead access and obviating the need for source code. By selecting hardware features relevant to the unpacking process, we train machine learning classifiers in a supervised learning manner. In our study, we investigate the use of hardware performance counters for the classification purpose of binary packers. Our findings indicate that when configured to alleviate nondeterminism, Hardware Performance Counters possess the capability and potential to classify prevalent packers that are readily accessible.

The Computer Journal, Volume 62, Issue 12, December 2019, Pages 1734-1747, doi: 10.1093/comjnl/bxz033. In the originally published version of this manuscript, requested author amendments were inadvertently omitted prior to publishing and have since been corrected. The Publisher would like to apologise for any inconvenience caused to the reader.

Abstract Malware brings a big security threat on the Internet today. With the great increasing malware attacks. Behavior-based detection approaches are one of the major method to detect zero-day malware. Such approaches often use API calls to represent the behavior of malware. Unfortunately, behavior-based approaches suffer from behavior obfuscation attacks. In this paper, we propose a novel malware detection approach that is both effective and efficient. First, we abstract the API call to object operation. And then we generate the object operation dependency graph based on these object operations. Our approach use dependency graph to represent the behavior of malware. The evaluation results show that our approach can provide a complete resistance to all types of behavior obfuscation attacks, and outperforms existing behavior-based approaches in terms of better effectiveness and efficiency.

Binary packing, encoding binary code prior to execution and decoding them at run time, is the most common obfuscation adopted by malware authors to camouflage malicious code. Especially, most packers recover the original code by going through a set of "written-then-executed" layers, which renders determining the end of the unpacking increasingly difficult. Many generic binary unpacking approaches have been proposed to extract packed binaries without the prior knowledge of packers. However, the high runtime overhead and lack of anti-analysis resistance have severely limited their adoptions. Over the past two decades, packed malware is always a veritable challenge to anti-malware landscape. This paper revisits the long-standing binary unpacking problem from a new angle: packers consistently obfuscate the standard use of API calls. Our in-depth study on an enormous variety of Windows malware packers at present leads to a common property: malware's Import Address Table (IAT), which acts as a lookup table for dynamically linked API calls, is typically erased by packers for further obfuscation; and then unpacking routine, like a custom dynamic loader, will reconstruct IAT before original code resumes execution. During a packed malware execution, if an API is invoked through looking up a rebuilt IAT, it indicates that the original payload has been restored. This insight motivates us to design an efficient unpacking approach, called BinUnpack. Compared to the previous methods that suffer from multiple "written-then-executed" unpacking layers, BinUnpack is free from tedious memory access monitoring, and therefore it introduces very small runtime overhead. To defeat a variety of ever-evolving evasion tricks, we design BinUnpack's API monitor module via a novel kernel-level DLL hijacking technique. We have evaluated BinUnpack's efficacy extensively with more than 238K packed malware and multiple Windows utilities. BinUnpack's success rate is significantly better than that of existing tools with several orders of magnitude performance boost. Our study demonstrates that BinUnpack can be applied to speeding up large-scale malware analysis.