Abstract: With the widely use of search engines, the impact Web robots have on the Web sites should not be ignored. After analyzing the navigational patterns of Web robots, a new algorithm based on content classification and statistics is proposed. The experiment s

Key words: Search engine, Web robot, Content classification, Detection, Web log