计算机科学 ›› 2010, Vol. 37 ›› Issue (1): 251-254.

• 人工智能 • 上一篇    下一篇

基于自适应中文分词和近似SVM的文本分类算法

冯永,李华,钟将,叶春晓   

  1. (重庆大学计算机学院 重庆400030)
  • 出版日期:2018-12-01 发布日期:2018-12-01
  • 基金资助:
    本文受重庆市自然科学基金(2008BB2183),中国博士后科学基金(20080440699),国家社会科学基金(ACA07004—08)资助。

Text Classification Algorithm Based on Adaptive Chinese Word Segmentation and Proximal SVM

FENG Yong,LI Hua,ZHONG Jiang,YE Chun-xiao   

  • Online:2018-12-01 Published:2018-12-01

摘要: 中文分词的难点在于处理歧义和识别未登录词,传统字典的匹配算法很大程度上是依靠字典的代表性而无法有效地识别新词,特别是对于各种行业领域的知识管理。基于二元统计模型的分词算法能很好地适应不同的语料信息,且时间和精度都能满足文本知识管理的应用需要。近似支持向量机是将问题归结成仅含线性等式约束的二次规划问题,该算法的时间复杂度和空间复杂度比传统SVM算法的均有降低。在利用自适应分词算法进行分词的基础上,再利用近似支持向量机进行文本分类。实验表明,该方法能够自动适应行业领域的知识管理,且满足文本知识管理对训练时间敏感和需要处理大量文本的苛刻环境要求,从而具备较大的实用价值。

关键词: 自适应中文分词,近似支持向量机,文本分类,知识管理

Abstract: New words recognition and ambiguity resolving are key problems in Chinese word segmentation. The result of traditional dictionary-based matching algorithm largely depends on the representative of the dictionary so that it can not recognize new words effectively, especially in some professional domains. Chinese word segmentation method in this dissertation is based on 2-gram statistical model and can meet the rectuirements of application in accuracy and efficiency respectively. PSVM takes classification as a linear equality quadratic programming problem. This dissertation describes a text classification algorithm based on adaptive Chinese word segmentation and PSVM, which has faster training speed and smaller memory requirements advantages. Several data sets of experiments showed that the classification algorithm can automatically adapt to knowledge management of some professional domains and has better classfication performance under the condition of timcsensitive.

Key words: Adaptive Chinese word segmentation, Proximal support vector machines, Text classification, Knowledge management

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!