计算机科学 ›› 2022, Vol. 49 ›› Issue (6): 319-325.doi: 10.11896/jsjkx.210600123

• 人工智能 • 上一篇    下一篇

基于BERT-GRU-ATT模型的中文实体关系分类

赵丹丹1,2, 黄德根1, 孟佳娜2, 董宇2, 张攀2   

  1. 1 大连理工大学计算机科学与技术学院 辽宁 大连 116024
    2 大连民族大学计算机科学与工程学院 辽宁 大连 116600
  • 收稿日期:2021-06-16 修回日期:2021-10-22 出版日期:2022-06-15 发布日期:2022-06-08
  • 通讯作者: 孟佳娜(mengjn@dlnu.edu.cn)
  • 作者简介:(zhaodd@dlnu.edu.cn)
  • 基金资助:
    国家科技创新2030-“新一代人工智能”重大项目(2020AAA008004);国家自然科学基金(U1936109,61876031);辽宁省教育厅科研经费(LJYT201906)

Chinese Entity Relations Classification Based on BERT-GRU-ATT

ZHAO Dan-dan1,2, HUANG De-gen1, MENG Jia-na2, DONG Yu2, ZHANG Pan2   

  1. 1 School of Computer Science and Technology,Dalian University of Technology,Dalian,Liaoning 116024,China
    2 School of Computer Science and Engineering,Dalian Minzu University,Dalian,Liaoning 116600,China
  • Received:2021-06-16 Revised:2021-10-22 Online:2022-06-15 Published:2022-06-08
  • About author:ZHAO Dan-dan,born in 1975,postgra-duate,associate professor,is a member of China Computer Federation.Her main research interests include nature language processing and information extraction.
    MENG Jia-na,born in 1972,Ph.D,professor,is a member of China Computer Federation.Her main research interests include nature language processing,transfer learning and sentiment ana-lysis.
  • Supported by:
    National Key Research and Development Program of China(2020AAA008004),National Natural Science Foundation of China(U1916109,61876031) and Scientific Research Fund of Liaoning Provincial Education Department (LJYT201906).

摘要: 实体关系分类作为自然语言处理的基础任务,对知识图谱、智能问答、语义网构建等任务都起到了非常关键的作用。文中构建了BERT-GRU-ATT模型,以进行中文实体关系分类。为消除中文分词歧义对实体关系分类的影响,引入预训练模型BERT作为嵌入层,以较好地获得汉字的上下文信息;再通过双向门控循环单元捕获实体在句子中的长距离依赖,通过自注意力机制加强对关系分类贡献明显的字的权重,从而获得较好的实体关系分类结果。为了丰富中文实体关系分类语料,将SemEval2010_Task8英文实体关系评测语料翻译为中文1),该模型在此翻译语料上取得了75.46%的F1值,说明了所提模型的有效性。此外,所提模型在SemEval2010-task8英文数据集上F1值达到了80.55%,证明该模型对英文语料具有一定的泛化能力。

关键词: 门控循环单元, 预训练模型, 中文实体关系分类, 自注意力机制

Abstract: As the basic task of natural language processing,entity relations classification plays a critical role in tasks such as knowledge graphs,intelligent question answering,semantic web construction and so on.This paper constructs the BERT-GRU-ATT model to classify Chinese entity relations.In order to eliminate the influence of Chinese word segmentation ambiguity on entity relations classification,the pre-training model BERT(bi-directional encoder representations from transformers) is introduced as the embedding layer to better obtain the context information of Chinese characters.Then gate recurrent unit(GRU) is used to capture the long-distance dependence of entities in sentences and self-attention mechanism(ATT) is used to strengthen the weight of characters that contribute significantly to relations classification,so as to obtain better results of entity relations classification.In order to enlarge the Chinese entity relations classification corpus,we translate the SemEval2010_Task8 English entity relations evaluation corpus into Chinese.The model achieves an F1 value of 75.46% on this translation corpus,which shows the effectiveness of the proposed model.In addition,the model achieves an F1 of 80.55%on the SemEval2010-Task8 English dataset,which proves that the model has certain generalization ability to English corpus.

Key words: Chinese entity relations classification, Gate recurrent unit, Pre-training model, Self-attention mechanism

中图分类号: 

  • TP391
[1] XU J,ZHANG Z X,WU Z X.Review on Techniques of Entity Relation Extraction[J].Data Analysis and Knowledge Discovery,2008,24(8):18-23.
[2] LIU Q,LI Y,DUAN H,et al.Knowledge Graph ConstructionTechniques[J].Journal of Computer Research and Development,2016,53(3):582-600.
[3] LIU Y J.Research on Automatic Extraction of Chinese NamedEntities and Entity Relations[D].Zhengzhou:Zhengzhou University,2019.
[4] E H H,ZHANG W J,XIAO S Q,et al.Survey of Entity Relationship Extraction Based on Deep Learning[J].Journal of Software,2019,30(6):1793-1818.
[5] LI D M,ZHANG Y,LI D Y,et al.Review of Entity Relation Extraction Methods[J].Journal of Computer Research and Development,2020,57(7):1424-1448.
[6] DENG B,FAN X Z,YANG L G.Entity Relation ExtractionMethod Using Semantic Pattern[J].Computer Engineering,2007,33(10):212-214.
[7] CHE W X,LIU T,LI S.Automatic Entity Relation Extraction[J].Journal of Chinese Information Processing,2005,19(2):2-7.
[8] GUO X Y,HE T T,HU X H,et al.Chinese Named Entity Relation Extraction Based on Syntactic and Semantic Features[J].Journal of Chinese Information Processing,2014,28(6):183-189.
[9] GAO J P,ZHANG H,ZHAO X J,et al.Evolutionary Relation Extraction for Domain Knowledge in Wikipedia[J].Chinese Journal of Computers,2016,39(10):2088-2101.
[10] LIU K B,LI F,LIU L,et al.Implementation of a Kernel-Based Chinese Relation Extraction System[J].Journal of Computer Research and Development,2007,44(8):1406-1411.
[11] GUO J Y,CHEN P,YU Z T,et al.Domain Specific Chinese Semantic Relation Extraction Based on Composite Kernel[J].Journal of Chinese Information Processing,2016,30(1):24-30.
[12] YU H H,QIAN L H,ZHOU G D,et al.Chinese Semantic Rela-tion Extraction Based on Unified Syntactic and Entity Semantic Tree[J].Journal of Chinese Information Processing,2010,24(5):7-23.
[13] PETRONI F,CORRO L,GEMULLA R.Core:Context-awareopen relation extraction with factorization machines[C]//Proceedings of the 2015 Conference on Empirical Methodes in Na-tural Language Processing.Lisbon,Portugal,2015:1763-1773.
[14] QIU L K,ZANG Y.ZORE:A syntax-based system for Chinese open relation extraction[C]//Proceedings of the 2014 Confe-rence on Empirical Methods in Natural Language Processing.Doha,Qatar,2014:1870-1880.
[15] QIN B,LIU A A,LIU T.Unsupervised Chinese Open EntityRelation Extraction[J].Journal of Computer Research and Development,2015,52(5):1029-1035.
[16] ZENG D,LIU K,LAI S,et al.Relation classification via convolutional deep neural network[C]//Proceedings of the 25th International Conference on Computational Linguistics:Technical Papers(COLING 2014).Dublin,Ireland,2014:2335-2344.
[17] XU K,FENG Y,HUANG S,et al.Semantic relation classification via convolutional neural networks with simple negative sampling[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.Lisbon,Portugal,2015:941-949.
[18] SANTOS C N D,XIANG B,ZHOU B.Classifying relations by ranking with convolutional neural networks[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.Beijing,China,2015:132-137.
[19] YE H,CHAO W H,LUO Z C,et al.Jointly extracting relations with class ties via effective deep ranking[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Vancouver,Canada,2017:1810-1820.
[20] BAI F,RITTER A.Structured minimally supervised learning for neural relation extraction[C]//Proceedings of the 2019 Confe-rence of the North American Chapter of the Association for Computational Linguistics.Minneapolis,Minnesota,2019:3057-3069.
[21] GAO D,PENG D L,LIU C.Entity Relation Extraction Based on CNN in Large-scale Text Data[J].Journal of Chinese Computer Systems 2018,39(5):1021-1026.
[22] SUN J D,GU X S,LI Y,et al.Chinese entity relation extraction algorithms based on COAE2016 datasets[J].Journal of Shandong University(Natural Science) 2017,52(9):7-12,18.
[23] SOCHER R,HUVAL B,MANNING C D,et al.Semantic Compositionality through Recursive Mtrix-Vector Spaces[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Prcessing and Computational Natural Language Learning.Stroudsburg,PA,UA,2012:1201-1211.
[24] LIN Y K,SHEN S Q,LIU Z Y,et al.Neural relation extraction with selective attention over instances[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Berlin,Germany,2016:2124-2133.
[25] CHEN L,TIMOTHY M,DMITRIY D,et al.Self-training im-proves recurrent neural networks performance for temporal relation extraction[C]//Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis.Brussels,Belgium,2018:165-176.
[26] XU Y,MOU L L,LI G,et al.Classifying relations via long short term memory networks along shortest dependency paths[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.Lisbon,Portugal,2015:1785-1794.
[27] NGOC T V,ADEL H,GUPTA P,et al.Combining recurrentand convolutional neural networks for relation classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics.San Diego,California,2016:534-539.
[28] SCHLICHTKRULL M S,KIPF T,BLOEM P,et al.Modeling relational data with graph convolutional networks[C]//Proceedings of the Europen semantic Web Conference on.Heraklion,Crete,Greece,2018:593-607.
[29] ZHANG Y H,QI P,MANNING C D.Graph convolution over pruned dependency trees improves relation extraction[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.Brussels,Belgium,2018:2205-2215.
[30] SONG L F,ZHANG Y,WANG Z G,et al.N-ary relation extraction using graph state LSTM[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proces-sing.Brussels,Belgium,2018:2226-2235.
[31] GUO Z J,ZHANG Y,LU W.Attention guided graph convolutional networks for relation extraction[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence,Italy,2019:241-251.
[32] CHEN Y,ZHENG D Q,ZHAO T J.Chinese relation extraction based on Deep Belief Nets[J].Journal of Software,2012,23(10):2572-2585.
[1] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[2] 金方焱, 王秀利.
融合RACNN和BiLSTM的金融领域事件隐式因果关系抽取
Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM
计算机科学, 2022, 49(7): 179-186. https://doi.org/10.11896/jsjkx.210500190
[3] 张嘉淏, 刘峰, 齐佳音.
一种基于Bottleneck Transformer的轻量级微表情识别架构
Lightweight Micro-expression Recognition Architecture Based on Bottleneck Transformer
计算机科学, 2022, 49(6A): 370-377. https://doi.org/10.11896/jsjkx.210500023
[4] 刘硕, 王庚润, 彭建华, 李柯.
基于混合字词特征的中文短文本分类算法
Chinese Short Text Classification Algorithm Based on Hybrid Features of Characters and Words
计算机科学, 2022, 49(4): 282-287. https://doi.org/10.11896/jsjkx.210200027
[5] 胡艳丽, 童谭骞, 张啸宇, 彭娟.
融入自注意力机制的深度学习情感分析方法
Self-attention-based BGRU and CNN for Sentiment Analysis
计算机科学, 2022, 49(1): 252-258. https://doi.org/10.11896/jsjkx.210600063
[6] 侯宏旭, 孙硕, 乌尼尔.
蒙汉神经机器翻译研究综述
Survey of Mongolian-Chinese Neural Machine Translation
计算机科学, 2022, 49(1): 31-40. https://doi.org/10.11896/jsjkx.210900006
[7] 汤世征, 张岩峰.
DragDL:一种易用的深度学习模型可视化构建系统
DragDL:An Easy-to-Use Graphical DL Model Construction System
计算机科学, 2021, 48(8): 220-225. https://doi.org/10.11896/jsjkx.200900045
[8] 刘文洋, 郭延哺, 李维华.
识别关键蛋白质的混合深度学习模型
Identifying Essential Proteins by Hybrid Deep Learning Model
计算机科学, 2021, 48(8): 240-245. https://doi.org/10.11896/jsjkx.200700076
[9] 王胜, 张仰森, 陈若愚, 向尕.
基于细粒度差异特征的文本匹配方法
Text Matching Method Based on Fine-grained Difference Features
计算机科学, 2021, 48(8): 60-65. https://doi.org/10.11896/jsjkx.200700008
[10] 徐少伟, 秦品乐, 曾建朝, 赵致楷, 高媛, 王丽芳.
基于多级特征和全局上下文的纵膈淋巴结分割算法
Mediastinal Lymph Node Segmentation Algorithm Based on Multi-level Features and Global Context
计算机科学, 2021, 48(6A): 95-100. https://doi.org/10.11896/jsjkx.200700067
[11] 王习, 张凯, 李军辉, 孔芳, 张熠天.
联合自注意力和循环网络的图像标题生成
Generation of Image Caption of Joint Self-attention and Recurrent Neural Network
计算机科学, 2021, 48(4): 157-163. https://doi.org/10.11896/jsjkx.200300146
[12] 黄欣, 雷刚, 曹远龙, 陆明名.
基于深度学习的交互式问答研究综述
Review on Interactive Question Answering Techniques Based on Deep Learning
计算机科学, 2021, 48(12): 286-296. https://doi.org/10.11896/jsjkx.210100209
[13] 后同佳, 周良.
基于双向GRU神经网络和注意力机制的中文船舶故障关系抽取方法
Chinese Ship Fault Relation Extraction Method Based on Bidirectional GRU Neural Network and Attention Mechanism
计算机科学, 2021, 48(11A): 154-158. https://doi.org/10.11896/jsjkx.210100215
[14] 周小诗, 张梓葳, 文娟.
基于神经网络机器翻译的自然语言信息隐藏
Natural Language Steganography Based on Neural Machine Translation
计算机科学, 2021, 48(11A): 557-564. https://doi.org/10.11896/jsjkx.210100015
[15] 李杭, 李维华, 陈伟, 杨仙明, 曾程.
基于Node2vec和知识注意力机制的诊断预测
Diagnostic Prediction Based on Node2vec and Knowledge Attention Mechanisms
计算机科学, 2021, 48(11A): 630-637. https://doi.org/10.11896/jsjkx.210300070
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!