Abstract: In view of the low efficiency of traditional malicious program detection and the lack of automatic analysis of malicious programs,this paper studied to use recurrent neural networks to detect and classify malicious programs in deep learning environment.First,the QEMU is used to capture the API and its parameter sequence that are called when the malicious program runs,after the behavior abstraction,the characteristic sequence of the malicious program is formed.Then the feature sequence is mapped to a fixed length word vector by using a logarithmic bilinear model (HLBL),and these word vectors are synthesized into an input matrix of a recursive neural network (RNN).Through the training of the recursive neural network model,a multi-layer semantic aggregation model of malicious programs is established to complete the classification detection of malicious programs.The experimental data show that the recursive neural network model can detect malicious program effectively in the classification of malicious program detection.Compared with the traditional machine learning algorithm,its detection rate has increased by 17%.In particular,when the concept of tensors is introduced,after using the Recursive Neural Tensor Network (RNTN) model,the detection rate is increased by 7% compared to the RNN model by reducing the overall number of parameters and the amount of calculations.The experimental data fully show that the recursive neural network model can complete the detection and classification of malicious programs in big data environment.

Key words: Hierarchical log-bilinear language model, Multi-levelsemantic aggregate model, Quick emulator, Recursive neural network, Word vector