中文“非多字词错误”自动校对方法研究

doi:10.11896/j.issn.1002-137X.2016.10.038

摘要/Abstract

摘要： 针对目前中文文本中的“非多字词错误”自动校对方法的不足,提出了一种模糊分词的“非多字词错误”自动查错和自动校对方法。首先利用精确匹配算法与中文串模糊相似度算法对中文文本进行精确切分和模糊全切分,建立词图；然后利用改进的语言模型对词图进行最短路径求解,得到分词结果,实现“非多字词错误”的自动发现和自动纠正。实验测试集是由2万行领域问答系统日志语料构成,共包含664处“非多字词错误”。实验表明,所提方法能有效发现“非多字词错误”,包括由于汉字替换、缺字、多字引起的“非多字词错误”,该方法的查错召回率达到75.9%,查错精度达到85%。所提方法是一种将查错与纠错融于一体的方法。

关键词: 非词错误,非多字词错误,模糊匹配,词图

Abstract: Aiming at the insufficiency existing in current automatic proofreading method of Chinese non-multi-character word error,a method based on fuzzy segmentation of automatic error-detecting and automatic proofreading for ‘non-multi-character word error’ in Chinese texts was proposed.Firstly,we used exact matching algorithm for exact word segmentation and fuzzy matching algorithm for Chinese strings to full fuzzy segmentation,and built the word graph.In order to get the best segmentation results,a method based on an improved language model is used to solve the shortest path.After that,the ‘non-multi-character word errors’ in Chinese texts are achieved to be automatic error-detected and automatic proofreaded.The test set has 20000 sentences of domain question answering log which include 664 non-multi-character word errors.Experiments show that the method proposed in our paper can detect the ‘non-multi-character word error’ effectively including character substitution errors,deletion errors and insertion errors.The error-detection recall rate is 75.9% and the error-detection accuracy rate is 85％.The proposed method combines automatic error-detecting and automatic error-correction together.

Key words: Non-word error,Non-multi-character word error,Fuzzy matching,Word graph

刘亮亮,曹存根. 中文“非多字词错误”自动校对方法研究[J]. 计算机科学, 2016, 43(10): 200-205. https://doi.org/10.11896/j.issn.1002-137X.2016.10.038

LIU Liang-liang and CAO Cun-gen. Study of Automatic Proofreading Method for Non-multi-character Word Error in Chinese Text[J]. Computer Science, 2016, 43(10): 200-205. https://doi.org/10.11896/j.issn.1002-137X.2016.10.038

参考文献

[1] Kukich K.Techniques for automatically correcting words in text[J].ACM Computing Surveys (CSUR),1992,24(4):377-439
[2] Zhang Yang-sen,Cao Yuan-da,Yu Shi-wen.A Hybrid Model of Combining Rule-based and Statistics-based Approaches for Automatic Detecting Errors in Chinese Text[J].J ournal of Chinese Information Processing,2006,20:1-7(in Chinese) 张仰森,曹元大,俞士汶.基于规则与统计相结合的中文文本自动查错模型与算法[J].中文信息学报,2006,20:1-7
[3] Shi De-sheng,Wang Liang-zhi,Chen Zhi-da,et al.A Statistics-based Approach for Automatic Detecting Errors in Chinese Text[J].Computer and Communications,1992,8:19-26(in Chinese) 施得胜,王良志,陈志达,等.基于统计的中文错字侦测法[J].电脑与通讯,1992,8:19-26
[4] Zhang Zhao-huang.Automatic Error Detection and Correction of Chinese Text[J].Communications of COLIPS,1994,4(2):143-149(in Chinese) 张照煌.中文错别字自动订正方法初探[J].Communications of COLIPS,1994,4(2):143-149
[5] Zhang L,Zhou M,Huang C,et al.Multifeature-based approach to automatic error detection and correction of Chinese text[C]∥Proceedings of the First Workshop on Natural Language Processing and Neural Networks.2000:2744-2748
[6] Ma Jin-shan,Zhang Yu,Liu Ting,et al.Detecting Chinese Text Errors Based on Trigram and Dependency Parsing[J].Journal of the China Society for Scientific and Technical Information,2005,23(6):723-728(in Chinese) 马金山,张宇,刘挺,等.利用三元模型及依存分析查找中文文本错误[J].情报学报,2005,23(6):723-728
[7] Zhang Yang-sen,Cao Yuan-da,Yu Shi-wen.A Hybrid Model of Combining Rule-based and Statistics-based Approaches for Automatic Detecting Errors in Chinese Text[J].J ournal of Chinese Information Processing,2006,20(4):1-7(in Chinese) 张仰森,曹元大,俞士汶.基于规则与统计相结合的中文文本自动查错模型与算法[J].中文信息学报,2006,20(4):1-7
[8] Zhang Yang-sen,Ding Bing-qing.Automatic Errors Detecting of Chinese Texts Based on the Bi-neighborship[J].Journal of Chinese Information Processing,2001,15(3):36-43(in Chinese) 张仰森,丁冰青.基于二元接续关系检查的字词级自动查错方法[J].中文信息学报,2001,15(3):36-43
[9] Zhang Lei,Sun Mao-song,Zhou Ming,et al.Automatic Chinese Text Error Correction Approach Based on Fast Approximate Chinese Word-Matching Algorithm[C]∥Procceding of the 3^rd World Congress on Intelligent Control and Automation.2000:2739-2743(in Chinese) 张磊,孙茂松,周明,等.基于快速模糊词匹配算法的中文自动校对方法[C]∥第三届全球智能控制与自动化大会.2000:2739-2743
[10] Liu Liang-liang,Wang Shi,Wang Dong-sheng,et al.AutomaticText Error Detection in Domain Question Answering[J].Journal of Chinese Information Processing,2013,27(3):77-83(in Chinese) 刘亮亮,王石,王东升,等.领域问答系统中的文本错误自动发现方法[J].中文信息学报,2013,27(3):77-83
[11] 朱德熙.语法讲义[M].商务印书馆,1982
[12] Zhang Hua-ping,Liu Qun.Model of Chinese Words Rough Segmentation Based on N-Shortest-Paths Method[J].Journal of Chinese Information Processing,2002,6(5):1-7(in Chinese) 张华平,刘群.基于N-最短路径方法的中文词语粗分模型[J].中文信息学报,2002,6(5):1-7
[13] 丰强泽,曹存根.语音查询中的辨音方法[M].Google Patents.CN Patent App.CN02,0,272
[14] 王石,王卫民,符建辉.一种汉字字形认知相似度计算方法[M].Google Patents,28.CN Patent App.CN 201,0,205,7
[15] Wang Si-li,Zhang Hua-ping,Wang Bin.Research of Optimization on Double-Array Trie and its Application[J].Journal of Chinese Information Processing,2006,20:24-30(in Chinese) 王思力,张华平,王斌.双数组Trie树算法优化及其应用研究[J].中文信息学报,2006,20:24-30

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed