中文工艺规范文本分词语料的构建与研究

doi:10.11896/jsjkx.221200070

Abstract

Abstract: Chinese word segmentation is a basic task for process specification text processing,which has a critical impact on downstream tasks such as process knowledge graphs and intelligent Q&A systems.One of the challenges faced by word segmentation of process specification texts is the lack of high-quality annotated corpus,especially word segmentation specifications for special language phenomena such as terms,noun phrases,process parameters,and quantifiers.This paper formulates a special word segmentation specification for the process specification text,collects and annotates a word segmentation corpus for Chinese process specification text(WS-MPST),including 11900 sentences and 255160 words,and the consistency of word segmentation by 4 annotators achieves 95.25%.On the WS-MPST corpus,the famous BiLSTM-CRF and BERT-CRF models are tested,and the F1 values achieves 92.61% and 93.69% respectively.Experimental results show that it is necessary to construct a special word segmentation corpus for process specification test.The in-depth analysis of experimental results reveals that the out-of-vocabularywords and the words which contain Chinese and non-Chinese characters are difficultto segment in process specification texts,which provides some guidance for future word segmentation research in process specification texts and related fields.

Key words: Chinese word segmentation, Process specification text, Word segmentation specification, Word segmentation corpus, Word segmentation model

CLC Number:

TP391

WANG Peiyan, ZHANG Yingxin, FU Xiaoqiang, CHEN Jiaxin, XU Nan, CAI Dongfeng. Construction and Research of Chinese Word Segmentation Corpus of Process Specification Text[J].Computer Science, 2023, 50(11A): 221200070-6.

References

[1]China National Committee for Terminology in Science andTech-nology.Mechanical Engineering Terms(Second Edition)[M].Beijing:Science Press,2021.
[2]GU X H,BAO J S,LV C F.Assembly semantic informationmodeling based on knowledge graph[J].Aeronautical Manufacturing’ Technology,2021,64(4):74-81.
[3]ZHU J N,LIANG Y Q,GU F,et al.Design of knowledge question-answering system for mechanical intelligentmanufacturing based on deep learning[J].Computer Integrated Making System,2019,25(5):1161-1168.
[4]CHEN Z Y,BAO J S,ZHENG X H,et al.Semantic recognition method of assembly process based on LSTM[J].Computer Integrated Making System,2021,27(6):1583-1593.
[5]HUANG C L,ZHAO H.Chinese Word Segmentation:A DecadeReview[J].Journal of Chinese Information Processing,2007(3):8-19.
[6]ZHAO H,CAI D,HUANG C L,et al.Chinese Word Segmentation:Another Decade Review(2007-2017)[J].arXiv:1901.06079,2019.
[7]EMERSON T.The Second International Chinese Word Segmentation Bakeoff[C]//Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.Jeju Island,Korea,2005:123-133.
[8]XUE N W,XIA F,CHIOU F D,et al.The Penn Chinese Treebank:PhraseStructure Annotation of a Large Corpus[J].Natural Language Engineering,2005,11(2):207.
[9]HUANG K Y,HUANG D G,LIU Z,et al.A Joint Multiple Criteria Model in Transfer Learning for Cross-domain Chinese Word Segmentation[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.2020:3873-3882.
[10]TIAN Y,SONG Y,XIA F,et al.Improving Chinese Word Segmentation with Wordhood Memory Networks[C]//Proceedings of the 58th Annual Meeting of the Association for Computa-tional Linguistics.2020.
[11]LIU Y,TIAN Y,CHANG T H,et al.Exploring Word Segmentation and Medical Concept Recognition for Chinese Medical Texts[C]//Proceedings of the 20th Workshop on Biomedical Language Processing.2021:213-220.
[12]LIU Y,ZHANG Y.Unsupervised Domain Adaptation for Joint Segmentation and POS-Tagging[C]//Proceedings of CoLING 2012.2012:745-754.
[13]QIU L K,ZHANG Y.Word Segmentation for Chinese Novels[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence.2015:2440-2446.
[14]ZHANG G P,LIU D S,YIN B S,et al.Research on Chinese Word Segmentation for Patent Documents[J].Journal of Chinese Information Processing,2010,24(3):112-116.
[15]ZHANG J,ZHANG H C,ZHAI D S,et al.Research of theWord Segmentation for Chinese Patent Claims[J].New Technology of Library and Information Service,2014(9):91-98.
[16]YUE J Y,XU J A,ZHANG Y J.Chinese Word Segmentation for Patent Documents[J].Journal of Peking University,2013,49(1):159-164.
[17]GB/T 13715-1992,Contemporary Chinese language word se-gmentation specification for information processing[S].Beijing:China Standard Press,1992.
[18]HRIPCSAK G,ROTHSCHILD A.Agreement,the F-measure,and Reliability in Information Retrieval[J].Journal of the American medical informatics association,2005,12(3):296-298.
[19]HUANG Z,WEI X,KAI Y.Bidirectional LSTM-CRF Modelsfor Sequence Tagging[J].arXiv:1508.01991,2015.
[20]LAFFERTY J,MCCALLUM A,PEREIRA F.Conditional random fields:Probabilistic models for segmenting and labeling sequence data[C]//Proceedings of ICML’01.2001:282-289.
[21]MA J,GANCHEV K,WEISS D,et al.State-of-the-art Chinese Word Segmentation with Bi-LSTMs[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:4902-4908.
[22]GONG J,CHEN X,GUI T,et al.Switch-LSTMs for Multi-Cri-teria Chinese Word Segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019:6457-6464.
[23]DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.2019:4171-4186.
[24]SUN X,ZHANG Y Z,MATSUZKI T,et al.A discriminative latent variable Chinese segmenter with hybrid word/character information[C]//Proceedings of Human Language Technologies:The Annual Conference of the North American Chapter of the Association for Computational Linguistics.2009:56-64.
[25]GB/T 24735-2009.Numbering Method for Machine-BuildingTechnological Documentation[S].Beijing:China Standard Press,2009.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Construction and Research of Chinese Word Segmentation Corpus of Process Specification Text

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 8

Metrics

Comments

Recommended 0

[1]	LIU Kai, ZHANG Hong-jun, CHEN Fei-qiong. Name Entity Recognition for Military Based on Domain Adaptive Embedding [J]. Computer Science, 2022, 49(1): 292-297.
[2]	GONG Fa-ming,ZHU Peng-hai. Word Segmentation Based on Adaptive Hidden Markov Model in Oilfield [J]. Computer Science, 2018, 45(6A): 97-100.
[3]	LI Jin-ting, HOU Hong-xu, WU Jing, WANG Hong-bin and FAN Wen-ting. Effect of Preprocessing on Corpus of Mongolian-Chinese Statistical Machine Translation [J]. Computer Science, 2017, 44(10): 259-264.
[4]	WANG Qing-song and WEI Ru-yu. Bayesian Chinese Spam Filtering Method Based on Phrases [J]. Computer Science, 2016, 43(4): 256-259.
[5]	LIANG Xi-tao and GU Lei. Active Learning in Chinese Word Segmentation Based on Nearest Neighbor [J]. Computer Science, 2015, 42(6): 228-232.
[6]	FENG Yong,LI Hua,ZHONG Jiang,YE Chun-xiao. Text Classification Algorithm Based on Adaptive Chinese Word Segmentation and Proximal SVM [J]. Computer Science, 2010, 37(1): 251-254.
[7]	. [J]. Computer Science, 2007, 34(9): 174-175.
[8]	XIA Xin-Song,　 XIAO Jian-Guo　（Institute of Computer Science and Technology of Peking University, Beijing 100084）. [J]. Computer Science, 2006, 33(3): 160-164.