基于深度学习的机器阅读理解综述

doi:10.11896/j.issn.1002-137X.2019.07.002

摘要/Abstract

摘要： 阅读理解能力是人类智能中最关键的能力之一,而机器阅读理解作为自然语言处理领域皇冠上的明珠,一直是该领域的研究焦点。近年来,随着深度学习方法的快速发展,机器阅读理解技术获得了长足的进步。首先,对基于深度学习的机器阅读理解技术的研究背景和发展历史进行了概述;然后,详细介绍了词向量、注意力机制以及答案预测这三大关键技术的研究进展;在此基础上,分析了目前机器阅读理解研究所面临的问题;最后,对机器阅读理解技术的未来发展趋势进行了展望。

关键词: 词向量, 机器阅读理解, 深度学习, 注意力机制, 自然语言处理

Abstract: Natural language processing is the key to achieving artificial intelligence.Machine reading comprehension,as the crown jewel in the field of natural language processing,has always been the focus of research in the field.With the rapid development of deep learning and neural network in recent years,machine reading comprehension has made great progress.Firstly,the research background and development history of machine reading comprehension were introduced.Then,by reviewing the important progress in the development of word vector,attention mechanism and answer prediction,the problems in recent research related to machine reading comprehension were proposed.Finally,the outlook of machine reading comprehension was discussed.

Key words: Attention mechanism, Deep learning, Machine reading comprehension, Natural language processing, Word vector

中图分类号:

TP391

李舟军,王昌宝. 基于深度学习的机器阅读理解综述[J]. 计算机科学, 2019, 46(7): 7-12. https://doi.org/10.11896/j.issn.1002-137X.2019.07.002

LI Zhou-jun,WANG Chang-bao. Survey on Deep-learning-based Machine Reading Comprehension[J]. Computer Science, 2019, 46(7): 7-12. https://doi.org/10.11896/j.issn.1002-137X.2019.07.002

参考文献

[1]HERMANN K M,KOCISKY T,GREFENSTETTE E,et al. Teaching machines to read and comprehend[C]∥Advances in Neural Information Processing Systems.Cambridge:MIT Press,2015:1693-1701.
[2]RAJPURKAR P,ZHANG J,LOPYREV K,et al.SQuAD: 100 000+ Questions for Machine Comprehension of Text[C]∥Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:ACL,2016:2383-2392.
[3]NGUYEN T,ROSENBERG M,SONG X,et al.MS MARCO:A human generated machine reading comprehension dataset [DB/OL].[2018-08-28].https://arxiv.org/abs/1611.09268.
[4]HIRSCHMAN L,LIGHT M,BRECK E,et al.Deep read:A reading comprehension system[C]∥Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics.Stroudsburg:ACL,1999:325-332.
[5]RILOFF E,THELEN M.A rule-based question answering system for reading comprehension tests[C]∥Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems-Volume 6.Stroudsburg:ACL,2000:13-19.
[6]POON H,CHRISTENSEN J,DOMINGOS P,et al.Machine reading at the university of washington[C]∥Proceedings of the NAACL HLT 2010 First International Workshop on Forma-lisms and Methodology for Learning by Reading.Association for Computational Linguistics.Stroudsburg:ACL,2010:87-95.
[7]RICHARDSON M,BURGES C J C,RENSHAW E.Mctest:A challenge dataset for the open-domain machine comprehension of text[C]∥Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.Stroudsburg:ACL,2013:193-203.
[8]KADLEC R,SCHMID M,BAJGAR O,et al.Text understan- ding with the attention sum reader network [DB/OL].[2018-08-28].https://arxiv.org/abs/1603.01547.
[9]CHEN D,BOLTON J,MANNING C D.A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task[C]∥Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Stroudsburg:ACL,2016:2358-2367.
[10]CUI Y,CHEN Z,WEI S,et al.Attention-over-Attention Neural Networks for Reading Comprehension[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Stroudsburg:ACL,2017:593-602.
[11]DHINGRA B,LIU H,YANG Z,et al.Gated-Attention Readers for Text Comprehension[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vo-lume 1:Long Papers).Stroudsburg:ACL,2017:1832-1846.
[12]WANG S,JIANG J.Machine comprehension using match-lstm and answer pointer [DB/OL].[2018-08-28].https://ar-xiv.org/abs/1608.07905.
[13]WANG S,JIANG J.Learning Natural Language Inference with LSTM[C]∥Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Stroudsburg:ACL,2016:1442-1451.
[14]VINYALS O,FORTUNATO M,JAITLY N.Pointer networks[C]∥Advances in Neural Information Processing Systems.Cambridge:MIT Press,2015:2692-2700.
[15]SEO,MINJOON,et al.Bidirectional attention flow for machine comprehension [DB/OL].[2018-08-28].https://arxiv.org/abs/1611.01603.
[16]XIONG C,ZHONG V,SOCHER R.Dynamic coattention network for question answering:U.S.Patent Application 15/421,193[P].2018-05-10.
[17]WANG W,YANG N,WEI F,et al.Gated self-matching networks for reading comprehension and question answering[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Stroudsburg:ACL,2017:189-198.
[18]JOSHI M,CHOI E,WELD D,et al.TriviaQA:A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Stroudsburg:ACL,2017:1601-1611.
[19]HE W,LIU K,LYU Y,et al.DuReader:a Chinese Machine Reading Comprehension Dataset from Real-world Applications[DB/OL].[2018-08-28].https://arxiv.org/abs/1711.05073.
[20]TAN C,WEI F,YANG N,et al.S-net:From answer extraction to answer generation for machine reading comprehension [DB/OL].[2018-08-28].https://arxiv.org/abs/1706.04815.
[21]CLARK C,GARDNER M.Simple and Effective Multi-Para- graph Reading Comprehension[C]∥Meeting of the Association for Computational Linguistics.Stroudsburg:ACL,2018:845-855.
[22]WANG Y,LIU K,LIU J,et al.Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification[C]∥Meeting of the Association for Computational Linguistics.Stroudsburg:ACL,2018:1918-1927.
[23]KOCISKY T,SCHWARZ J,BLUNSOM P,et al.The Narra- tiveQA Reading Comprehension Challenge[J].Transactions of the Association for Computational Linguistics,2018,6:317-328.
[24]DEERWESTER S,DUMAIS S T,FURNAS G W,et al.Indexing by latent semantic analysis[J].Journal of the American society for Information Science,1990,41(6):391-407.
[25]LUND K,BURGESS C.Producing high-dimensional semantic spaces from lexical co-occurrence[J].Behavior Research Me-thods,Instruments,& Computers,1996,28(2):203-208.
[26]ROHDE D L T,GONNERMAN L M,PLAUT D C.An improved model of semantic similarity based on lexical co-occurrence[J].Communications of the ACM,2006,8(627-633):116.
[27]BENGIO Y,DUCHARME R,VINCENT P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2003,3:1137-1155.
[28]MIKOLOV T,SUTSKEVER I,CHEN K,et al.Distributed representations of words and phrases and their compositionality[C]∥dAdvancesf in Neural Information Processing Systems.Cambridge:MIT Press,2013:3111-3119.
[29]MIKOLOV T,CHEN K,CORRADO G,et al.Efficient estimation of word representations in vector space[J].arXiv:1301.3781,2013.
[30]PENNINGTON J,SOCHER R,MANNING C.Glove:Global vectors for word representation[C]∥Proceedings of the 2014 Conference on Empirical Methods in Natural Language Proces-sing (EMNLP).Stroudsburg:ACL,2014:1532-1543.
[31]MELAMUD O,GOLDBERGER J,DAGAN I.context2vec: Learning generic context embedding with bidirectional lstm[C]∥Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning.Stroudsburg:ACL,2016:51-61.
[32]MCCANN B,BRADBURY J,XIONG C,et al.Learned in translation:Contextualized word vectors[C]∥Advances in Neural Information Processing Systems.Cambridge:MIT Press,2017:6294-6305.
[33]PETERS M,NEUMANN M,IYYER M,et al.Deep Contextua- lized Word Representations[C]∥Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies(Vo-lume 1:Long Papers).Stroudsburg:ACL,2018:2227-2237.
[34]RENSINK R A.The dynamic representation of scenes[J].Vi- sual cognition,2000,7(1／2／3):17-42.
[35]MNIH V,HEESS N,GRAVES A.Recurrent models of visual attention[C]∥Advances in Neural Information Processing Systems.Stroudsburg:ACL,2014:2204-2212.
[36]BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate [DB/OL].[2018-08-28].https://arxiv.org/abs/1409.0473.
[37]CUI Y,LIU T,CHEN Z,et al.Consensus Attention-based Neural Networks for Chinese Reading Comprehension[C]∥Proceedings of COLING 2016,the 26th International Conference on Computational Linguistics:Technical Papers.Pisa:ACM,2016:1777-1786.
[38]CUI Y,CHEN Z,WEI S,et al.Attention-over-Attention Neural Networks for Reading Comprehension[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Stroudsburg:ACL,2017:593-602.
[39]WANG S,JIANG J.Machine comprehension using match-lstm and answer pointer [DB/OL].[2018-08-28].https://arxiv.org/abs/1608.07905.

相关文章 15

[1]	周芳泉, 成卫青. 基于全局增强图神经网络的序列推荐 Sequence Recommendation Based on Global Enhanced Graph Neural Network 计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[2]	戴禹, 许林峰. 基于文本行匹配的跨图文本阅读方法 Cross-image Text Reading Method Based on Text Line Matching 计算机科学, 2022, 49(9): 139-145. https://doi.org/10.11896/jsjkx.220600032
[3]	周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[4]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5]	熊丽琴, 曹雷, 赖俊, 陈希亮. 基于值分解的多智能体深度强化学习综述 Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization 计算机科学, 2022, 49(9): 172-182. https://doi.org/10.11896/jsjkx.210800112
[6]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[7]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[8]	朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[9]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[10]	闫佳丹, 贾彩燕. 基于双图神经网络信息融合的文本分类方法 Text Classification Method Based on Information Fusion of Dual-graph Neural Network 计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[11]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[12]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[13]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[14]	汪鸣, 彭舰, 黄飞虎. 基于多时间尺度时空图网络的交通流量预测模型 Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction 计算机科学, 2022, 49(8): 40-48. https://doi.org/10.11896/jsjkx.220100188
[15]	侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed