基于BERT的中文命名实体识别方法

摘要/Abstract

摘要： 针对传统的机器学习算法对中文实体识别准确率低、高度依赖特征设计以及领域自适应能力差的问题,提出了基于BERT的神经网络方法进行命名实体识别。首先,利用大规模未标注语料对BERT进行训练,获取文本抽象特征;然后,利用BiLSTM神经网络获取序列化文本的上下文抽象特征;最后,通过CRF进行序列解码标注,提取出相应的实体。该方法结合BERT和BiLSTM-CRF模型对中文实体进行识别,以无需添加任何特征的方式在1998上半年人民日报数据集上取得了94.86%的F1值。实验表明,该方法提升了实体识别的准确率、召回率及F1值,验证了该方法的有效性。

关键词: BERT, BiLSTM, 命名实体识别, 条件随机场, 序列标注

Abstract: In order to solve the problems of low accuracy of traditional machine learning algorithms in Chinese entity recognition,high dependence on feature design and poor adaptability in the field,a recurrent neural network method based on bidirectional encoder representation from transformers was proposed for named entity recognition.Firstly,the BERT is trained by large-scale unlabeled corpus to obtain the abstract features of the text.Then the BiLSTM neural network is used to obtain the contextual features of the serialized text.Finally,the corresponding entities are extracted by sequence labeling with CRF.The method combines the BERT and BiLSTM-CRF models for Chinese entity recognition,and has obtained the F1 value of 94.86% on the People's Daily data set in the first half of 1998 without adding any features.Experiments show that this method improves the accuracy,recall rate and F1 value of entity recognition,indicating the effectiveness of this method.

Key words: BERT, BiLSTM, Conditional random fields, Named entity recognition, Sequence labeling

中图分类号:

TP391

王子牛, 姜猛, 高建瓴, 陈娅先. 基于BERT的中文命名实体识别方法[J]. 计算机科学, 2019, 46(11A): 138-142. https://doi.org/

WANG Zi-niu, JIANG Meng, GAO Jian-ling, CHEN Ya-xian. Chinese Named Entity Recognition Method Based on BERT[J]. Computer Science, 2019, 46(11A): 138-142. https://doi.org/

参考文献

[1]SANG,ERIK F,KIM T,et al.Introduction to the CoNLL-2003 Shared Task:Language-Independent Named Entity Recognition[J].IEEE TransactionsonWireless Communications,2003,21(8):142-147.
[2]BENGIO Y,SCHWENK H,SENÉCAL J S.Neural Probabilistic Language Models[J].Journal of Machine Learning Research,2001,3(6):1137-1155.
[3]赵晓凡,赵丹,刘永革.利用CRF实现中文人名性别的自动识别[J].微电子学与计算机,2011,28(10):122-128.
[4]林广和,张绍武,林鸿飞.基于细粒度词表示的命名实体识别研究[J].中文信息学报,2018,32(11):62-71.
[5]LUO G,HUANG X J,LIN C Y,et al.Joint named entity recognition and disambiguation[C]∥Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.2015:1030-1038.
[6]ZHANG L,ZHANG Y.Big Data Analysis by Infinite Deep Neural Networks [J].Journal of Computer Research and Development,2016,53(1):68-79.
[7]ZHANG S,WANG X.Automatic Recognition of Chinese Or-ganization Name Based on Conditional Random Fields[C]∥International Conference on Natural Language Processing and Knowledge Engineering,2007.IEEE,2007:229-233.
[8]BORTHWICK A E.A maximum entropy approach to named entity recognition[M].New York:New York University,1999.
[9]程健一,关毅,何彬.基于SVM和CRF双层分类器的英文电子病历去隐私化[J].智能计算机与应用,2016,6(6):17-19.
[10]COLLOBERT R,WESTON J,BOTTOU L,et al.Natural language processing (almost) fromscratch[J].Journal of Machine Learning Research,2011,12(1):2493-2537.
[11]GRAVES A.Long Short-Term Memory[M]∥Supervised Sequence Labelling with Recurrent Neural Networks.Springer Berlin Heidelberg,2012:1735-1780.
[12]殷昊,徐健.基于字词融合特征的微博情绪识别方法[J].计算机科学,2018,45(S2):105-109.
[13]CHO K,VAN MERRIENBOER B,GULCEHRE C,et al.Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[J].arXiv:1406.1078v3,2014.
[14]HUANG Z,XU W,YU K.BidirectionalLSTM-CRF modelsfor sequencetagging[J].arXiv:1508.01991,2015.
[15]LAMPLE G,BALLESTEROS M,SUBRAMAN S,et al.Neural architectures for namedentity recognition[C]∥Proceedings of NAACL-HLT.2016:260-270.
[16]CHIU J P,NICHOLS E.Namedentity recognition with bidirectional LSTM-CNNs[J].Transactions of the Association for Computational Linguistics,2015,4(10):357-370.
[17]FENG Y H,HONG Y U,SUN G,et al.Named Entity Recognition Method Based on BLSTM[J].Computer Science,2018,45(2):261-268.
[18]SHEN Y,YUN H,LIPTON Z C,et al.Deep Active Learning for Named Entity Recognition[J].arXiv:1707.05928v3,2018.
[19]JEFFREY P,SOCHER R,MANNING C D.GloVe:Global Vectors for Word Representation[C]∥Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).2014:1532-1543.
[20]DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[J].arXiv:1810.04805v1,2018.
[21]VASWANI A,NOAM S,et al.Attention Is All You Need[J].arXiv:1706.03762v5,2017.
[22]HOCHREITER S,SCHMIDHUBER J.Long Short-term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[23]GRAVES A,MOHAMED A R,HINTON G.Speech Recognition with Deep Recurrent Neural Networks[C]∥IEEE International Conference on Acoustics,Speech and Signal Processing.2013:6645-6649.
[24]PETERS M E,NEUMANN M.Deep contextualized word representations[J].arXiv:1802.05365v2,2018.
[25]RADFORD A,NARASIMHAN K.Improving Language Understanding by Generative Pre-Training[J/OL].https://s3-us-west2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_unders-tanding_paper.pdf.

相关文章 15

[1]	于家畦, 康晓东, 白程程, 刘汉卿. 一种新的中文电子病历文本检索模型 New Text Retrieval Model of Chinese Electronic Medical Records 计算机科学, 2022, 49(6A): 32-38. https://doi.org/10.11896/jsjkx.210400198
[2]	杜晓明, 袁清波, 杨帆, 姚奕, 蒋祥. 军事指控保障领域命名实体识别语料库的构建 Construction of Named Entity Recognition Corpus in Field of Military Command and Control Support 计算机科学, 2022, 49(6A): 133-139. https://doi.org/10.11896/jsjkx.210400132
[3]	康雁, 吴志伟, 寇勇奇, 张兰, 谢思宇, 李浩. 融合Bert和图卷积的深度集成学习软件需求分类 Deep Integrated Learning Software Requirement Classification Fusing Bert and Graph Convolution 计算机科学, 2022, 49(6A): 150-158. https://doi.org/10.11896/jsjkx.210500065
[4]	余本功, 张子薇, 王惠灵. 一种融合多层次情感和主题信息的TS-AC-EWM在线商品排序方法 TS-AC-EWM Online Product Ranking Method Based on Multi-level Emotion and Topic Information 计算机科学, 2022, 49(6A): 165-171. https://doi.org/10.11896/jsjkx.210400238
[5]	郭雨欣, 陈秀宏. 融合BERT词嵌入表示和主题信息增强的自动摘要模型 Automatic Summarization Model Combining BERT Word Embedding Representation and Topic Information Enhancement 计算机科学, 2022, 49(6): 313-318. https://doi.org/10.11896/jsjkx.210400101
[6]	丁锋, 孙晓. 基于注意力机制和BiLSTM-CRF的消极情绪意见目标抽取 Negative-emotion Opinion Target Extraction Based on Attention and BiLSTM-CRF 计算机科学, 2022, 49(2): 223-230. https://doi.org/10.11896/jsjkx.210100046
[7]	刘凯, 张宏军, 陈飞琼. 基于领域适应嵌入的军事命名实体识别 Name Entity Recognition for Military Based on Domain Adaptive Embedding 计算机科学, 2022, 49(1): 292-297. https://doi.org/10.11896/jsjkx.201100007
[8]	程思伟, 葛唯益, 王羽, 徐建. BGCN:基于BERT和图卷积网络的触发词检测 BGCN:Trigger Detection Based on BERT and Graph Convolution Network 计算机科学, 2021, 48(7): 292-298. https://doi.org/10.11896/jsjkx.200500133
[9]	董哲, 邵若琦, 陈玉梁, 翟维枫. 基于BERT和对抗训练的食品领域命名实体识别 Named Entity Recognition in Food Field Based on BERT and Adversarial Training 计算机科学, 2021, 48(5): 247-253. https://doi.org/10.11896/jsjkx.200800181
[10]	陈明豪, 祝跃飞, 芦斌, 翟懿, 李玎. 基于Attention-CNN的加密流量应用类型识别 Classification of Application Type of Encrypted Traffic Based on Attention-CNN 计算机科学, 2021, 48(4): 325-332. https://doi.org/10.11896/jsjkx.200900155
[11]	张栋, 陈文亮. 基于上下文相关字向量的中文命名实体识别 Chinese Named Entity Recognition Based on Contextualized Char Embeddings 计算机科学, 2021, 48(3): 233-238. https://doi.org/10.11896/jsjkx.191200074
[12]	余诗媛, 郭淑明, 黄瑞阳, 张建朋, 苏珂. 嵌套命名实体识别研究进展 Overview of Nested Named Entity Recognition 计算机科学, 2021, 48(11A): 1-10. https://doi.org/10.11896/jsjkx.201100165
[13]	陈德, 宋华珠, 张娟, 周泓林. 融合BERT和记忆网络的实体识别 Entity Recognition Fusing BERT and Memory Networks 计算机科学, 2021, 48(10): 91-97. https://doi.org/10.11896/jsjkx.200900015
[14]	龚扣林, 周宇, 丁笠, 王永超. 基于BiLSTM模型的漏洞检测 Vulnerability Detection Using Bidirectional Long Short-term Memory Networks 计算机科学, 2020, 47(5): 295-300. https://doi.org/10.11896/jsjkx.190800046
[15]	唐国强,高大启,阮彤,叶琪,王祺. 融入语言模型和注意力机制的临床电子病历命名实体识别 Clinical Electronic Medical Record Named Entity Recognition Incorporating Language Model and Attention Mechanism 计算机科学, 2020, 47(3): 211-216. https://doi.org/10.11896/jsjkx.190200259

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed