Abstract: Compared with other types of malware,botnets have recently been adopted by hackers for their resiliency against take-down efforts.Besides being harder to take down,modern botnets tend to be stealthier in the way they perform malicious activities by using the infected computer,making current detection approaches ineffective.Given the malicious activities botnets can realize,detection and mitigation of botnet threats are imperative.In this paper,we presented a novel approach for botnet detection,called distributed real-time botnet detection algorithm.It uses Spark engine,where Netflow related data are correlated as the host Netflow graph structure and the host access chain structure,and a feature extraction method based on the Spark Streaming is leveraged for exacting implicit characteristics.Meanwhile,this paper established distributed BotScanner detection system based on the Spark Streaming,which is a distributed real-time steam processing engine.We trained BotScanner system on the five representative bot families and evaluated BotScanner on simulated network traffic and real-world network traffic.The experimental results show that the BotScanner is able to detect bots in network traffic without the need of deep packet inspection,and achieves high detection rates with very few false positives.When the traffic data from the Internet service provider are very large,the BotScanner is able to detect botnets in real-time by adding the compute nodes,and BotScanner has approximate linear speedup.It proves the feasibility of Applying Spark Streaming engine to distributed botnet detection.

Key words: Big data,Botnet,Real-time detection,Spark streaming