We describe Instruction-Set Randomization (ISR), a general approach for safeguarding systems against any type of code-injection attack. We apply Kerckhoffs' principle to create OS process-specific randomized instruction sets (e.g., machine instructions) of the system executing potentially vulnerable software. An attacker who does not know the key to the randomization algorithm will inject code that is invalid for that (randomized) environment, causing a runtime exception. Our approach is applicable to machine-language programs and scripting and interpreted languages. We discuss three approaches (protection for Intel x86 executables, Perl scripts, and SQL queries), one from each of the above categories. Our goal is to demonstrate the generality and applicability of ISR as a protection mechanism. Our emulator-based prototype demonstrates the feasibility ISR for x86 executables and should be directly usable on a suitably modified processor. We demonstrate how to mitigate the significant performance impact of emulation-based ISR by using several heuristics to limit the scope of randomized (and interpreted) execution to sections of code that may be more susceptible to exploitation. The SQL prototype consists of an SQL query-randomizing proxy that protects against SQL injection attacks with no changes to database servers, minor changes to CGI scripts, and with negligible performance overhead. Similarly, the performance penalty of a randomized Perl interpreter is minimal. Where the performance impact of our proposed approach is acceptable (i.e., in an already-emulated environment, in the presence of programmable or specialized hardware, or in interpreted languages), it can serve as a broad protection mechanism and complement other security mechanisms.

We present e-NeXSh, a novel security approach that utilises kernel and LIBC support for efficiently defending systems against process-subversion attacks. Such attacks exploit vulnerabilities in software to override its program control-flow and consequently invoke system calls, causing out-of-process damage. Our technique defeats such attacks by monitoring all LIBC function and system-call invocations, and validating them against process-specific information that strictly prescribes the permissible behaviour for the program (unlike general sandboxing techniques that require manually maintained, explicit policies, we use the program code itself as a guideline for an implicit policy). Any deviation from this behaviour is considered malicious, and we halt the attack, limiting its damage to within the subverted process. We implemented e-NeXSh as a set of modifications to the Linux-2.4.18-3 kernel and a new user-space shared library (e-NeXSh.so). The technique is transparent, requiring no modifications to existing libraries or applications. e-NeXSh was able to successfully defeat both code-injection and LIBC-based attacks in our effectiveness tests. The technique is simple and lightweight, demonstrating no measurable overhead for select UNIX utilities, and a negligible 1.55% performance impact on the Apache Web server

We describe a new, general approach for safeguarding systems against any type of code-injection attack. We apply Kerckhoff's principle, by creating process-specific randomized instruction sets (e.g., machine instructions) of the system executing potentially vulnerable software. An attacker who does not know the key to the randomization algorithm will inject code that is invalid for that randomized processor, causing a runtime exception. To determine the difficulty of integrating support for the proposed mechanism in the operating system, we modified the Linux kernel, the GNU binutils tools, and the bochs-x86 emulator. Although the performance penalty is significant, our prototype demonstrates the feasibility of the approach, and should be directly usable on a suitable-modified processor (e.g., the Transmeta Crusoe).Our approach is equally applicable against code-injecting attacks in scripting and interpreted languages, e.g., web-based SQL injection. We demonstrate this by modifying the Perl interpreter to permit randomized script execution. The performance penalty in this case is minimal. Where our proposed approach is feasible (i.e., in an emulated environment, in the presence of programmable or specialized hardware, or in interpreted languages), it can serve as a low-overhead protection mechanism, and can easily complement other mechanisms.