关键词: 日志异常预测, 强化学习, 双重深度Q网络, 竞争深度Q网络, 日志序列图

Abstract: In the operation of large-scale software systems,anomalies such as hardware failures,malicious attacks,and module compatibility conflicts can easily lead to service interruptions and system crashes.Although logs provide key evidence for anomaly analysis,they still face multiple challenges:1)Traditional machine learning methods rely solely on event count vectors,ignoring the temporal dependencies and topological structures of log sequences;2)Deep learning methods are limited by the existing patterns in the training set,resulting in weak generalization capabilities,and they cannot model complex dependency relationships;3)Existing reinforcement learning methods suffer from issues such as Q-value overestimation and single reward mechanisms,failing to conduct forward-looking prediction and grading of abnormal sequences.To address the above challenges,a reinforcement learning-based solution is proposed.Firstly,it constructs a log sequence graph model based on a double deep Q-Network－under the condition of using only normal log samples and is guided by the experience replay mechanism and prior knowledge,the model structurally models the long-term dependencies of logs through sequence graphs,which breaks the reliance of existing reinforcement learning methods on a single state vector and solves the problems of Q-value overestimation and insufficient generalization.Secondly,it designs an anomaly prediction model based on a dueling deep Q-Network—through an adversarial learning mechanism and a multi-dimensional reward function,combined with an anomaly grading mechanism,the model achieves accurate prediction and grading of future abnormal sequences,which solves the limitation of traditional methods that “only detect but not predict” and provides sufficient response time for operation and maintenance.

Key words: Log anomaly prediction, Reinforcement learning, Double deep Q-Network, Dueling deep Q-Network, Log sequence graph