Beginning with the work of Forrest et al, several researchers have developed intrusion detection techniques based on modeling program behaviors in terms of system calls. A weakness of these techniques is that they focus on control flows involving system calls, but not their arguments. This weakness makes them susceptible to several classes of attacks, including attacks on security-critical data, race-condition and symbolic link attacks, and mimicry attacks. To address this weakness, we develop a new approach for learning dataflow behaviors of programs. The novelty in our approach, as compared to previous system-call argument learning techniques, is that it learns temporal properties involving the arguments of different system calls, thus capturing the flow of security-sensitive data through the program. An interesting aspect of our technique is that it can be uniformly layered on top of most existing control-flow models, and can leverage control-flow contexts to significantly increase the precision of dataflows captured by the model. This contrasts with previous system-call argument learning techniques that did not leverage control-flow information, and moreover, were focused on learning statistical properties of individual system call arguments. Through experiments, we show that temporal properties enable detection of many attacks that aren't detected by previous approaches. Moreover, they support formal reasoning about security assurances that can be provided when a program follows its dataflow behavior model, e.g., tar would read only files located within a directory specified as a command-line argument.

The ability to analyze and modify binaries is often very useful from a security viewpoint. Security operations one would like to perform on binaries include the ability to extract models of program behavior and insert inline reference monitors. Unfortunately, the existing manner in which binary code is packaged prevents even the simplest of analyses, such as distinguishing code from data, from succeeding 100 percent of the time. In this paper, we propose SELF, a security-enhanced ELF (Executable and Linking Format), which is simply ELF with an extra section added. The extra section contains information about (among other things) the address, size, and alignment requirements of each code and static data item in the program. This information is somewhat similar to traditional debugging information, but contains additional information specifically needed for binary analysis that debugging information lacks. It is also smaller, compatible with optimization, and less likely to facilitate reverse engineering, which we believe makes it practical for use with commercial software products. SELF approach has three key benefits. First, the information for the extra section is easy for compilers to provide, so little work is required on behalf of compiler vendors. Second, the extra section is ignored by default, so SELF binaries will run perfectly on all systems, including ones not interested in leveraging the extra information. Third, the extra section provides sufficient information to perform many security-related operations on the binary code. We believe SELF to be a practical approach, allowing many security analyses to be performed while not requiring major changes to the existing compiler infrastructure. An application example of the utility of SELF to perform address obfuscation (in which the addresses of all code and data items are randomized to defeat memory-error exploits) is presented.