Abstract: Buffer overflow attack has been a major security problem in recent years,where attackers utilize buffer overflow vulnerabilities to control other computers. As the vehicle of attack, Shellcode is the main target of buffer overflow attack detections. Now attackers tend to employ polymorphic techniques to encode Shellcode, which makes it harder for signature-based NIDS to detect it, This paper proposed a new method to detect the Shellcode executed under MS Windows, which integrates static analysis and dynamic execution techniques. It made some new principles of Shellcode detection, which enhance both the accuracy and performance of polymorphic Shellcode detection.Then a prototype system was implemented and tested. The test results on both the accuracy and performance are quite encouraging.

Key words: Shellcode,Static analysis,Dynamic execution