计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (11A): 221200070-6.doi: 10.11896/jsjkx.221200070

• 人工智能 • 上一篇    下一篇

中文工艺规范文本分词语料的构建与研究

王裴岩1, 张莹欣1, 付小强2, 陈佳欣1, 徐楠1, 蔡东风1   

  1. 1 沈阳航空航天大学人机智能研究中心 沈阳 110136
    2 中国商飞上海飞机制造有限公司航空制造技术研究所 上海 201324
  • 发布日期:2023-11-09
  • 通讯作者: 王裴岩(wangpy@sau.edu.cn)
  • 基金资助:
    辽宁省应用基础研究计划(2022JH2/101300248)

Construction and Research of Chinese Word Segmentation Corpus of Process Specification Text

WANG Peiyan1, ZHANG Yingxin1, FU Xiaoqiang2, CHEN Jiaxin1, XU Nan1, CAI Dongfeng1   

  1. 1 Human-Computer Intelligence Research Center,Shenyang Aerospace University,Shenyang 110136,China
    2 Aviation Manufacturing Technology Research Institute,COMAC Shanghai Aircraft Manufacturing,Shanghai 201324,China
  • Published:2023-11-09
  • About author:WANG Peiyan,born in 1983,Ph.D,senior engineer,is a member of China Computer Federation.His main research interests include natural language processing,machine learning and knowledge engineering.
  • Supported by:
    Applied Basic Research Program of Liaoning Province (2022JH2/101300248).

PDF (PC)

525

摘要： 中文分词是处理工艺规范文本的一项基本任务,并且在工艺知识图谱与智能问答等下游任务中发挥着重要作用。工艺规范文本分词面临的一个挑战是缺乏高质量标注的语料,特别是面向术语、名词短语、工艺参数、数量词等特殊语言现象的分词规范。文中面向工艺规范文本制定了专用分词规范,收集并标注了一个中文工艺规范文本分词语料(WS-MPST),含11 900个句子与255 160个词,4名标注者分词标注一致性达95.25%。在WS-MPST语料上对著名的BiLSTM-CRF与BERT-CRF模型进行了对比实验,F1值分别达到92.61%与93.69%。实验结果表明,构建专用的工艺规范分词语料是必要的。对实验结果的深入分析揭示了未登录词与中文非中文字符混合构成的词是工艺规范文本分词的难点,也为今后工艺规范文本及相关领域的分词研究提供了一定的指导。

关键词: 中文分词, 工艺规范文本, 分词规范, 分词语料, 分词模型

Abstract: Chinese word segmentation is a basic task for process specification text processing,which has a critical impact on downstream tasks such as process knowledge graphs and intelligent Q&A systems.One of the challenges faced by word segmentation of process specification texts is the lack of high-quality annotated corpus,especially word segmentation specifications for special language phenomena such as terms,noun phrases,process parameters,and quantifiers.This paper formulates a special word segmentation specification for the process specification text,collects and annotates a word segmentation corpus for Chinese process specification text(WS-MPST),including 11900 sentences and 255160 words,and the consistency of word segmentation by 4 annotators achieves 95.25%.On the WS-MPST corpus,the famous BiLSTM-CRF and BERT-CRF models are tested,and the F1 values achieves 92.61% and 93.69% respectively.Experimental results show that it is necessary to construct a special word segmentation corpus for process specification test.The in-depth analysis of experimental results reveals that the out-of-vocabularywords and the words which contain Chinese and non-Chinese characters are difficultto segment in process specification texts,which provides some guidance for future word segmentation research in process specification texts and related fields.

Key words: Chinese word segmentation, Process specification text, Word segmentation specification, Word segmentation corpus, Word segmentation model

中图分类号: 

  • TP391
[1]China National Committee for Terminology in Science andTech-nology.Mechanical Engineering Terms(Second Edition)[M].Beijing:Science Press,2021.
[2]GU X H,BAO J S,LV C F.Assembly semantic informationmodeling based on knowledge graph[J].Aeronautical Manufacturing’ Technology,2021,64(4):74-81.
[3]ZHU J N,LIANG Y Q,GU F,et al.Design of knowledge question-answering system for mechanical intelligentmanufacturing based on deep learning[J].Computer Integrated Making System,2019,25(5):1161-1168.
[4]CHEN Z Y,BAO J S,ZHENG X H,et al.Semantic recognition method of assembly process based on LSTM[J].Computer Integrated Making System,2021,27(6):1583-1593.
[5]HUANG C L,ZHAO H.Chinese Word Segmentation:A DecadeReview[J].Journal of Chinese Information Processing,2007(3):8-19.
[6]ZHAO H,CAI D,HUANG C L,et al.Chinese Word Segmentation:Another Decade Review(2007-2017)[J].arXiv:1901.06079,2019.
[7]EMERSON T.The Second International Chinese Word Segmentation Bakeoff[C]//Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.Jeju Island,Korea,2005:123-133.
[8]XUE N W,XIA F,CHIOU F D,et al.The Penn Chinese Treebank:PhraseStructure Annotation of a Large Corpus[J].Natural Language Engineering,2005,11(2):207.
[9]HUANG K Y,HUANG D G,LIU Z,et al.A Joint Multiple Criteria Model in Transfer Learning for Cross-domain Chinese Word Segmentation[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.2020:3873-3882.
[10]TIAN Y,SONG Y,XIA F,et al.Improving Chinese Word Segmentation with Wordhood Memory Networks[C]//Proceedings of the 58th Annual Meeting of the Association for Computa-tional Linguistics.2020.
[11]LIU Y,TIAN Y,CHANG T H,et al.Exploring Word Segmentation and Medical Concept Recognition for Chinese Medical Texts[C]//Proceedings of the 20th Workshop on Biomedical Language Processing.2021:213-220.
[12]LIU Y,ZHANG Y.Unsupervised Domain Adaptation for Joint Segmentation and POS-Tagging[C]//Proceedings of CoLING 2012.2012:745-754.
[13]QIU L K,ZHANG Y.Word Segmentation for Chinese Novels[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence.2015:2440-2446.
[14]ZHANG G P,LIU D S,YIN B S,et al.Research on Chinese Word Segmentation for Patent Documents[J].Journal of Chinese Information Processing,2010,24(3):112-116.
[15]ZHANG J,ZHANG H C,ZHAI D S,et al.Research of theWord Segmentation for Chinese Patent Claims[J].New Technology of Library and Information Service,2014(9):91-98.
[16]YUE J Y,XU J A,ZHANG Y J.Chinese Word Segmentation for Patent Documents[J].Journal of Peking University,2013,49(1):159-164.
[17]GB/T 13715-1992,Contemporary Chinese language word se-gmentation specification for information processing[S].Beijing:China Standard Press,1992.
[18]HRIPCSAK G,ROTHSCHILD A.Agreement,the F-measure,and Reliability in Information Retrieval[J].Journal of the American medical informatics association,2005,12(3):296-298.
[19]HUANG Z,WEI X,KAI Y.Bidirectional LSTM-CRF Modelsfor Sequence Tagging[J].arXiv:1508.01991,2015.
[20]LAFFERTY J,MCCALLUM A,PEREIRA F.Conditional random fields:Probabilistic models for segmenting and labeling sequence data[C]//Proceedings of ICML’01.2001:282-289.
[21]MA J,GANCHEV K,WEISS D,et al.State-of-the-art Chinese Word Segmentation with Bi-LSTMs[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:4902-4908.
[22]GONG J,CHEN X,GUI T,et al.Switch-LSTMs for Multi-Cri-teria Chinese Word Segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019:6457-6464.
[23]DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.2019:4171-4186.
[24]SUN X,ZHANG Y Z,MATSUZKI T,et al.A discriminative latent variable Chinese segmenter with hybrid word/character information[C]//Proceedings of Human Language Technologies:The Annual Conference of the North American Chapter of the Association for Computational Linguistics.2009:56-64.
[25]GB/T 24735-2009.Numbering Method for Machine-BuildingTechnological Documentation[S].Beijing:China Standard Press,2009.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发