Abstract: Intelligent attack path discovery is a key technology for automated penetration testing,but existing methods face the problems of exponential growth of state and action space and sparse rewards,which make the algorithm difficult to converge.To this end,an intelligent attack path discovery method(iPathD) based on hierarchical reinforcement learning is proposed.iPathD constructs the attack path discovery process as a layered Markov decision process to describe the upper-layer inter-host penetration path discovery and the lower-layer single-host internal attack path discovery,respectively.On this basis,an attack path discovery algorithm based on hierarchical reinforcement learning is proposed and implemented.Experimental results show that compared with the traditional method based on deep Q learning(DQN) and its improved algorithm,the iPathD path discovery method is faster and more effective.With the increase of the number of vulnerabilities in the host,the effect of iPathD is better,and it is suitable for large-scale network scenarios.

Key words: Penetration testing, Markov decision process, Hierarchical reinforcement learning, Attack path discovery, DQN algorithm