Computer Science ›› 2017, Vol. 44 ›› Issue (Z11): 460-463.doi: 10.11896/j.issn.1002-137X.2017.11A.098

Previous Articles     Next Articles

Research on Crawler Algorithm for Theme of Books

ZHANG Li-jing, ZENG Qing-tao, LI Ye-li, SUN Hua-yan and ZI Yun-fei   

  • Online:2018-12-01 Published:2018-12-01

Abstract: Aiming at the problem that the information crawling result of a book contains a lot of useless data,a kind of crawler algorithm was proposed,which is based on the book topic.The algorithm mainly consists of two parts,one part is based on the ODP (Open Directory System) dynamic keyword expansion method to describe the subject,the other part is the semantic extension of lexical entry based on VSM (Vector Space Model) topic correlation algorithm. The new algorithm,the VSM algorithm based on keywords and VSM algorithm based on ODP were analyzed through expe-riment.The result indicates that the precision and the recall rate of the new algorithm are higher than that of other two algorithms.

Key words: Focused crawler,ODP,VSM,Semantic extension

[1] 王聪睿.主题爬虫关键技术研究[D].石家庄:石家庄铁道大学,2015.
[2] 王良伟.面向垂直搜索引擎的主题爬虫方法研究[D].重庆:重庆大学,2013.
[3] 邱伟林.面向领域的垂直搜索引擎的研究与实现[D].大连:大连海事大学,2011.
[4] 刘建明.垂直搜索引擎中的主题爬虫技术研究[D].广州:广东工业大学,2013.
[5] 杜娟娟.主题爬虫算法的研究与实现[D].兰州:兰州交通大学,2013.
[6] 罗路天.垂直搜索引擎中主题网络爬虫算法的设计与研究[D].广州:广东工业大学,2016.
[7] LIU W J,DU Y J.An Improved Topic-specific Crawling Approach Based on Semantic Similarity Vector Space Model[J].Journal of Computational Information System,2012,8(20):8605-8612.
[8] 张燕平,刘超,曲永花.WCBVSM与SACA结合的文本分类模型[J].计算机工程与应用,2012,8(11):137-142.
[9] 苏喻,郑诚,马忠杰.基于语义的VSM模型改进[J].计算机应用与软件,2011,8(28):158-161.
[10] 吴麒,陈兴蜀,朱锴,等.基于ODP的上下文主题描述方法[J].电子学报,2012,1(40):2320-2323.
[11] 刘燕兵,谭建龙,郭莉.可动态增删关键词的串匹配算法[J].计算机工程与应用,2005,1(35):138-141.
[12] 张莉婧,李业丽,曾庆涛,等.基于改进TextRank的关键词抽取算法[J].北京印刷学院学报,2016,4(4):51-55.
[13] ZHU L,WANG G J,ZOU X C.A Study of Chinese Document Representation and Classification with Word2vec[C]∥9th International Symposium on Computational Intelligence and Design(ISCID).2016:298-302.
[14] LIU C H,LIU Q,LEE C H.Valence-arousal ratings prediction of Chinese words using similarity measures based on Word2Vec[C]∥International Conference on Asian Language Processing (IALP).2016:317-319.
[15] 游博.词语语义相关度计算研究[D].武汉:华中师范大学,2013.
[16] 李璐,张国印,等.基于SVM的主题爬虫技术研究[J].计算机科学,2015,2(2):118-122.
[17] 蒋华荣,郁雪.应用遗传算法优化子空间的SVM分类算法分类算法[J].计算机科学,2013,0(11):255-260,5.
[18] 王良芳.文本挖掘关键词提取算法的研究[D].杭州:浙江工业大学,2013.

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!