Software is commonly built from reusable components that provide desired functionalities. Although component reuse significantly improves software productivity, insecure component usage can lead to security vulnerabilities in client applications. For example, we noticed that widely-used IE-based browsers, such as IE Tab, do not enable important security features that IE enables by default, even though they all use the same browser components. This insecure usage renders these IE-based browsers vulnerable to the attacks blocked by IE. To our knowledge, this important security aspect of component reuse has largely been unexplored.

Dynamic loading of software components (e.g., libraries or modules) is a widely used mechanism for an improved system modularity and flexibility. Correct component resolution is critical for reliable and secure software execution. However, programming mistakes may lead to unintended or even malicious components being resolved and loaded. In particular, dynamic loading can be hijacked by placing an arbitrary file with the specified name in a directory searched before resolving the target component. Although this issue has been known for quite some time, it was not considered serious because exploiting it requires access to the local file system on the vulnerable host. Recently, such vulnerabilities have started to receive considerable attention as their remote exploitation became realistic. It is now important to detect and fix these vulnerabilities. In this paper, we present the first automated technique to detect vulnerable and unsafe dynamic component loadings. Our analysis has two phases: 1) apply dynamic binary instrumentation to collect runtime information on component loading (online phase), and 2) analyze the collected information to detect vulnerable component loadings (offline phase). For evaluation, we implemented our technique to detect vulnerable and unsafe component loadings in popular software on Microsoft Windows and Linux. Our evaluation results show that unsafe component loading is prevalent in software on both OS platforms, and it is more severe on Microsoft Windows. In particular, our tool detected more than 4,000 unsafe component loadings in our evaluation, and some can lead to remote code execution on Microsoft Windows.

The analysis of software similarity has many applications such as detecting code clones, software plagiarism, code theft, and polymorphic malware. Because often source code is unavailable and code obfuscation is used to avoid detection, there has been much research on developing effective models to capture runtime behavior to aid detection. Existing models focus on low-level information such as dependency or purely occurrence of function calls, and suffer from poor precision, poor scalability, or both. To overcome limitations of existing models, this paper introduces a precise and succinct behavior representation that characterizes high-level object-accessing patterns as regular expressions. We first distill a set of high-level patterns (the alphabet S of the regular language) based on two pieces of information: function call patterns to access objects and type state information of the objects. Then we abstract a runtime trace of a program P into a regular expression e over the pattern alphabet S to produce P's behavior signature. We show that software instances derived from the same code exhibit similar behavior signatures and develop effective algorithms to cluster and match behavior signatures. To evaluate the effectiveness of our behavior model, we have applied it to the similarity analysis of polymorphic malware. Our results on a large malware collection demonstrate that our model is both precise and succinct for effective and scalable matching and detection of polymorphic malware.

Dynamic loading of software components (e.g., libraries or modules) is a widely used mechanism for improved system modularity and flexibility. Correct component resolution is critical for reliable and secure software execution, however, programming mistakes may lead to unintended or even malicious components to be resolved and loaded. In particular, dynamic loading can be hijacked by placing an arbitrary file with the specified name in a directory searched before resolving the target component. Although this issue has been known for quite some time, it was not considered serious because exploiting it requires access to the local file system on the vulnerable host. Recently such vulnerabilities started to receive considerable attention as their remote exploitation became realistic; it is now important to detect and fix these vulnerabilities. In this paper, we present the first automated technique to detect vulnerable and unsafe dynamic component loadings. Our analysis has two phases: 1) apply dynamic binary instrumentation to collect runtime information on component loading (online phase); and 2) analyze the collected information to detect vulnerable component loadings (offline phase). For evaluation, we implemented our technique to detect vulnerable and unsafe DLL loadings in popular Microsoft Windows software. Our results show that unsafe DLL loading is prevalent and can lead to serious security threats. Our tool detected more than 1,700 unsafe DLL loadings in 28 widely used software and discovered serious attack vectors for remote code execution. Microsoft has opened a Microsoft Security Response Center (MSRC) case on our reported issues and is working with us and other affected software vendors to develop necessary patches.