Software obfuscation techniques have lost their effectiveness due to the rapid development of binary diffing techniques, which can achieve accurate function matching and identification. In this paper, we propose a new inter-procedural code obfuscation mechanism KHaos , 1 which moves the code across functions to obfuscate the function by using compilation optimizations. Three obfuscation primitives are proposed to separate, aggregate, and hide the function. They can be combined to enhance the obfuscation effect further. This article also reveals distinguishing factors on obfuscation and compiler optimization and presents novel observations to gain insights into the impact of actively utilizing compiler optimization in obfuscation. A prototype of KHaos is implemented and evaluated on a large number of real-world programs. Experimental results show that KHaos outperforms existing code obfuscations and can significantly reduce the accuracy rates of six state-of-the-art binary diffing techniques with lower runtime overhead.

In the field of binary code similarity detection (BCSD), when dealing with functions in binary form, the conventional approach is to identify a set of functions that are most similar to the target function. These similar functions often originate from the same source code but may differ due to variations in compilation settings. Such analysis is crucial for applications in the security domain, including vulnerability discovery, malware detection, software plagiarism detection, and patch analysis. Function inlining, an optimization technique employed by compilers, embeds the code of callee functions directly into the caller function. Due to different compilation options (such as O1 and O3) leading to varying levels of function inlining, this results in significant discrepancies between binary functions derived from the same source code under different compilation settings, posing challenges to the accuracy of state-of-the-art (SOTA) learning-based binary code similarity detection (LB-BCSD) methods. In contrast to function inlining, code extraction technology can identify and separate duplicate code within a program, replacing it with corresponding function calls. To overcome the impact of function inlining, this paper introduces a novel approach, CodeExtract. This method initially utilizes code extraction techniques to transform code introduced by function inlining back into function calls. Subsequently, it actively inlines functions that cannot undergo code extraction, effectively eliminating the differences introduced by function inlining. Experimental validation shows that CodeExtract enhances the accuracy of LB-BCSD models by 20% in addressing the challenges posed by function inlining.

Software obfuscation techniques can prevent binary diffing techniques from locating vulnerable code by obfuscating the third-party code, to achieve the purpose of protecting embedded device software. With the rapid development of binary diffing techniques, they can achieve more and more accurate function matching and identification by extracting the features within the function. This makes existing software obfuscation techniques, which mainly focus on the intra-procedural code obfuscation, no longer effective.

Binary diffing, which quantitatively measures the difference between given binaries, has been broadly used in critical security areas. Previous studies have been tackling the challenge of default compiler optimization, as it can affect binary representation but overlooked the exploration of non-default optimization settings, which can also significantly affect the accuracy of diffing. Recent research indicates a growing trend of compiling applications with non-default optimization settings to magnify binary code discrepancies, enabling them to evade detection by binary diffing tools. This paper takes the first step to systematically studying the resistance of compiler optimization (including default and non-default optimization settings) on binary diffing tasks. To this end, we construct a diverse and unique dataset, OPTBinary, with 3.6 million functions compiled from 514 optimization settings. Then, we propose OPTango, an innovative transformer-based multi-central representation learning approach, exploring the solution to build a compiler optimization-agnostic binary diffing tool. We conduct extensive experiments and benchmark OPTango with state-of-the-art binary diffing approaches. Evaluation results show that OPTango is more robust and significantly outperforms existing methods against both default and non-default compiler optimization.