Abstract: Alarm logs generated by network security equipment have a large number of repeated alarms,which impact real-time network situational threat analysis.In order to solve real-time accurate de-duplication problem of alarm logs,we proposed a method of duplicate removal on alert logs based on attributes hash.The method uses attribute hash for duplicate alarms quick detection and uses the hash table to solve the storage problem of a large number of non-repeating alarm logs at the same time.Conducted experiments results in the alarm log based on Darpa data set show that the method ensures lower time complexity,while deduplication accuracy rate can reach 95%.

Key words: Alert log,Repeat alert,Property hash