Abstract: Penetration path planning is the first step of penetration testing,which is important for the intelligent penetration testing.Existing studies on penetration path planning always model penetration testing as a full observable process,which is difficult to describe the actual penetration testing with partial observability accurately.With the wide application of reinforcement learning in penetration testing,this paper models the penetration testing as a partially observable Markov decision process to simulate the practical penetration testing accurately.In general,the full connection of policy network and evaluation network in PPO cannot extract features effectively in penetration testing with partial observability.This paper proposes an improved PPO algorithm RPPO,which integrating of full connection and long short term memory(LSTM) in the policy network and evaluation network.In addition,a new objective function updating is designed to improve the robustness and convergence.Experimental results show that,the proposed RPPO converges faster than A2C,PPO and NDSPI-DQN algorithms.Especially,the convergence iterations is reduced by 21.21%,28.64% and 22.85% respectively.Meanwhile RPPO gains more cumulative reward about 66.01%,58.61% and 132.64%,which is more suitable for larger-scale network environments with more than fifty hosts.

Key words: Penetration testing, Penetration path planning, Reinforcement learning, Proximal policy optimization, Long and short term memory networks