Computer Science ›› 2020, Vol. 47 ›› Issue (3): 211-216.doi: 10.11896/jsjkx.190200259

• Artificial Intelligence • Previous Articles     Next Articles

Clinical Electronic Medical Record Named Entity Recognition Incorporating Language Model and Attention Mechanism

TANG Guo-qiang,GAO Da-qi,RUAN Tong,YE Qi,WANG Qi   

  1. (School of information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China)
  • Received:2019-02-01 Online:2020-03-15 Published:2020-03-30
  • About author:TANG Guo-qiang,born in 1993,master.His main research interests include nature language processing and so on. GAO Da-qi,born in 1977,Ph.D,professor.His main research interests include pattern recognition and machine lear-ning.
  • Supported by:
    This work was supported by the National Key R&D Program of China (2018YFC0910500).

Abstract: Clinical Named Entity Recognition (CNER) aims to identify and classify named entity such as diseases,symptoms,exams, electronic health records,which is a fundamental and crucial task for clinical and translational research.The task is regarded as a sequence labeling problem.In recent years,deep neural network methods achieve significant success in named entity recognition.However,most of these algorithms do not take full advantages of the large amount of unlabeled data,and ignore the further features from the text.This paper proposed a model which combines language model and multi-head attention.First,chara-cter embeddings and a language model are trained from unlabeled clinical texts.Then,the labeling model are trained from labeled clinical texts.In specific use,the vector representation of the sentence is sent to a BiGRU and a pre-trained language model.This paper further concatenate the output of BiGRU and the features of language model.Afterwards,the outputs are fed to another BiGRU and multi-head attention module.Finally,a CRF layer is employed to predict the label sequence.Experimental results show that the proposed method which takes advantages of language model from the text and multi-head attention mechanism gets 91.34% of F1-score on CCKS-2017 Task2 benchmark dataset.

Key words: Clinical named entity recognition, Deep neural network, GRU, Language model, Multi-head attention

CLC Number: 

  • TP391
[2]GRIDACH M.Character-level neural network for biomedical named entity recognition[J].Journal of Biomedical Informatics,2017,70:85-91.
[3]HABIBI M,WEBER L,NEVES M,et al.Deep learning with word embeddings improves biomedical named entity recognition[J].Bioinformatics,2017,33(14):i37-i48.
[4]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]∥Advances in Neural Information Processing Systems.2017:5998-6008.
[5]FRIEDMAN C,ALDERSON P O,AUSTIN J H,et al.A general natural-language text processor for clinical radiology[J].J Am Med Inform Assoc,1994,1(2):161-174.
[6]ZENG Q T,GORYACHEV S,WEISS S,et al.Extracting principal diagnosis,co-morbidity and smoking status for asthma research:evaluation of a natural language processing system[J].BMC medical Informatics and Decision Making,2006,6(1):30.
[7]SAVOVA G K,MASANZ J J,OGREN P V,et al.Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES):architecture,component evaluation and applications[J].Journal of the American Medical Informatics Association Jamia,2010,17(5):507.
[8]RINDFLESCH T C,TANABE L,WEINSTEIN J N,et al.EDGAR:Extraction of Drugs,Genes And Relations from the Biomedical Literature[M]∥Biocomputing 2000.2014.
[9]SONG M,YU H,HAN W S.Developing a hybrid dictionary- based bio-entity recognition technique[J].BMC Medical Informatics and Decision Making,2015,15(1):S9.
[10]LEI J,TANG B,LU X,et al.A comprehensive study of named entity recognition in Chinese clinical text[J].Journal of the American Medical Informatics Association,2014,21(5):808-814.
[11]SETTLES B.Biomedical named entity recognition using conditional random fields and rich feature sets[C]∥Proceedings of the International Joint Workshop on Natural Language Proces-sing in Biomedicine and Its Applications.Association for Computational Linguistics,2004:104-107.
[12]SKEPPSTEDT M,KVIST M,NILSSON G H,et al.Automatic recognition of disorders,findings,pharmaceuticals and body structures from clinical text:An annotation and machine lear-ning study[J].Journal of Biomedical Informatics,2014,49:148-158.
[1] XIONG Luo-geng, ZHENG Shang, ZOU Hai-tao, YU Hua-long, GAO Shang. Software Self-admitted Technical Debt Identification with Bidirectional Gate Recurrent Unit and Attention Mechanism [J]. Computer Science, 2022, 49(7): 212-219.
[2] YANG Ya-hong, WANG Hai-rui. DDoS Attack Detection Method in SDN Environment Based on Renyi Entropy and BiGRU Algorithm [J]. Computer Science, 2022, 49(6A): 555-561.
[3] WEI Hui, CHEN Ze-mao, ZHANG Li-qiang. Anomaly Detection Framework of System Call Trace Based on Sequence and Frequency Patterns [J]. Computer Science, 2022, 49(6): 350-355.
[4] GAO Jie, LIU Sha, HUANG Ze-qiang, ZHENG Tian-yu, LIU Xin, QI Feng-bin. Deep Neural Network Operator Acceleration Library Optimization Based on Domestic Many-core Processor [J]. Computer Science, 2022, 49(5): 355-362.
[5] JIAO Xiang, WEI Xiang-lin, XUE Yu, WANG Chao, DUAN Qiang. Automatic Modulation Recognition Based on Deep Learning [J]. Computer Science, 2022, 49(5): 266-278.
[6] XIAO Ding, ZHANG Yu-fan, JI Hou-ye. Electricity Theft Detection Based on Multi-head Attention Mechanism [J]. Computer Science, 2022, 49(1): 140-145.
[7] FAN Hong-jie, LI Xue-dong, YE Song-tao. Aided Disease Diagnosis Method for EMR Semantic Analysis [J]. Computer Science, 2022, 49(1): 153-158.
[8] ZHANG Jin, DUAN Li-guo, LI Ai-ping, HAO Xiao-yan. Fine-grained Sentiment Analysis Based on Combination of Attention and Gated Mechanism [J]. Computer Science, 2021, 48(8): 226-233.
[9] ZHOU Xin, LIU Shuo-di, PAN Wei, CHEN Yuan-yuan. Vehicle Color Recognition in Natural Traffic Scene [J]. Computer Science, 2021, 48(6A): 15-20.
[10] PAN Fang, ZHANG Hui-bing, DONG Jun-chao, SHOU Zhao-yu. Aspect Sentiment Analysis of Chinese Online Course Review Based on Efficient Transformer [J]. Computer Science, 2021, 48(6A): 264-269.
[11] HU De-feng, ZHANG Chen-xi, WANG Shi-tao, ZHAO Qin-pei, LI Jiang-feng. Intelligent Prediction Model of Tool Wear Based on Deep Signal Processing and Stacked-ResGRU [J]. Computer Science, 2021, 48(6): 175-183.
[12] DING Ling, XIANG Yang. Chinese Event Detection with Hierarchical and Multi-granularity Semantic Fusion [J]. Computer Science, 2021, 48(5): 202-208.
[13] LIU Dong, WANG Ye-fei, LIN Jian-ping, MA Hai-chuan, YANG Run-yu. Advances in End-to-End Optimized Image Compression Technologies [J]. Computer Science, 2021, 48(3): 1-8.
[14] ZHANG Dong, CHEN Wen-liang. Chinese Named Entity Recognition Based on Contextualized Char Embeddings [J]. Computer Science, 2021, 48(3): 233-238.
[15] PAN Yu, ZOU Jun-hua, WANG Shuai-hui, HU Gu-yu, PAN Zhi-song. Deep Community Detection Algorithm Based on Network Representation Learning [J]. Computer Science, 2021, 48(11A): 198-203.
Full text



No Suggested Reading articles found!