计算机科学 ›› 2017, Vol. 44 ›› Issue (10): 276-282.doi: 10.11896/j.issn.1002-137X.2017.10.050
池云仙,赵书良,罗燕,高琳,赵骏鹏,李超
CHI Yun-xian, ZHAO Shu-liang, LUO Yan, GAO Lin, ZHAO Jun-peng and LI Chao
摘要： 在大数据时代,文本挖掘 面临特征的“高维-稀疏”问题,海量文本词汇与稀少关键特征间的矛盾导致了高时空复杂度和低效率等问题,严重制约了文本挖掘效率,因此在文本挖掘前进行有效的数据预处理至关重要。传统文本挖掘算法在数据预处理阶段只进行分词和去停用词操作。为提高性能,提出基于词频统计规律的文本数据预处理方法。首先,基于齐普夫定律和最大值法推导同频词数表达式；然后,基于同频词数表达式探究各频次词语在文中的分布规律,结果表明词频为1和2的词语与文档的关联度较低,但比重高达 2/3；最后,基于词频统计规律进行数据预处理,在预处理阶段去除低频词,减小特征维度。在公共数据集Reuters-21578和20-Newsgroups上进行的实验的结果表明,各频次词语的分布规律是正确的,基于词频统计规律的文本数据预处理方法在分类准确率、精确率、召回率以及F1度量值方面均有提升,运行时间明显降低,文本挖掘效率得到显著提高。
| [1] HAN J,FAN J,et al.Semanti-Enhanced Spatial Keyword Search[J].Journal of Computer Research and Development,2015,52(9):1954-1964.(in Chinese) 韩军,范举,等.一种语义增强的空间关键词搜索方法[J].计算机研究与发展,2015,52(9):1954-1964. [2] HU J,FAN J,LI G L,et al.Top-k Fuzzy Spatial KeywordSearch[J].Chinese Journal of Computers,2012,35(11):2237-2246.(in Chinese) 胡骏,范举,李国良,等.空间数据上Top-k关键词模糊查询算法[J].计算机学报,2012,35(11):2237-2246. [3] REN P J,CHEN Z M,et al.Search Result Diversification Combing Semantic and Temporal Intent[J].Chinese Journal of Computers,2015,38(10):2076-2091.(in Chinese) 任鹏杰,陈竹敏,等.一种综合语义和实效性意图的检索多样化方法[J].计算机学报,2015,38(10):2076-2091. [4] DING Z Y,JIA Y,et al.Survey of Data Mining for microblogs [J].Journal of Computer Reaserch and Development,2014,51(4):691-706.(in Chinese) 丁兆云,贾焰,等.微博数据挖掘综述[J].计算机研究与发展,2014,51(4):691-706. [5] SONG Q,NI J,WANG G.A fast clustering-based feature subset selection algorithm for high-dimensional data[J].IEEE Transactions on Knowledge and Data Engineering,2013,25(1):1-14. [6] ZHAO Z,HE X F,CAI D,et al.Graph Regularized Feature Selection with Data Reconstruction[J].IEEE Transactions on Knowledge and Data Engineering,2016,28(3):689-700. [7] ZIPF G K.Human behavior and the principle of least effort:an introduction to human ecology[M].Addison-Wesley Press,1949:23. [8] BOOTH A D.A law of occurrences for words of low frequency [J].Information and Control,1967,10(4):386-393. [9] EGGHE L.A new short proof of Naranan’s theorem,explaining Lotka’s law and Zipf’s law[J].Journal of the American Society for Information Science & Technology,2010,61(12):2581-2583. [10] CHAN P,HIJIKATA Y,NISHIDA S.Computing semantic relatedness using word frequency and layout information of wikipedia[C]∥ Proceedings of the 28th Annual ACM Symposium on Applied Computing.ACM,2013:282-287. [11] SURYASEN R,RANA M S.Content analysis and application of Zipf’s law in computer science literature [C]∥ 2015 4th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS).IEEE,2015:223-227. [12] GEORGE K Z.Human Behavior and the Principle of Least Effort:An Introduction to Human Ecology[M].New York:Addison-Wesley Press,1949:573-584. [13] 邱均平.文献计量学[M].科学技术文献出版社,1988:157. [14] BOOTH A D.A law of occurrences for words of low frequency [J].Information and Control,1967,10(4):386-393. [15] AGRAWAL R,GOLLAPUDI S,KENTHAPADI K.Enriching Textbooks Through Data Mining [C]∥ Proceedings of the First ACM Symposium on Computing for Development.2010:1-9. [16] AGRAWAL R,GOLLAPUDI S,KANNAN A,et al.Data mi-ning for improving textbooks[J].ACM SIGKDD Explorations Newsletter,2012,13(2):7-19. |
| No related articles found! |
|
||
首页