计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (6A): 220200120-8.doi: 10.11896/jsjkx.220200120

• 人工智能 • 上一篇    下一篇

知识增强的自然语言生成研究综述

梁明轩1,2, 王石2, 朱俊武1, 李阳1,2, 高翔1,2, 焦志翔1,2   

  1. 1 扬州大学信息工程学院 江苏 扬州 225000;
    2 中国科学院计算技术研究所 北京 100190
  • 出版日期:2023-06-10 发布日期:2023-06-12
  • 通讯作者: 王石(wangshi@ict.ac.cn)
  • 作者简介:(yzulmx@163.com)
  • 基金资助:
    国家242信息安全计划项目(2021A008);北京市科技新星计划交叉学科合作课题(Z191100001119014);国家重点研发计划重点专项(2017YFC1700300,2017YFB1002300);国家自然科学基金(61702234)

Survey of Knowledge-enhanced Natural Language Generation Research

LIANG Mingxuan1,2, WANG Shi2, ZHU Junwu1, LI Yang1,2, GAO Xiang1,2, JIAO Zhixiang1,2   

  1. 1 College of Information Engineering,Yangzhou University,Yangzhou,Jiangsu 225000,China;
    2 Institute of Computing Technology,Chinese Academy of Sciences,Beijing 100190,China
  • Online:2023-06-10 Published:2023-06-12
  • About author:LIANG Minxuan,born in 1998,master.His main research interest is natural language processing. WANG Shi,born in 1981, Ph.D,associate researcher,is a member of China Computer Federation.His main research interests include natural language processing semantic analysis and knowledge graph.
  • Supported by:
    National 242 Information Security Program(2021A008),Beijing NOVA Program (Z191100001119014),National Key Research and Development Program of China(2017YFC1700300,2017YFB1002300) and National Natural Science Foundation of China(61702234).

PDF (PC)

702

摘要： 自然语言生成(Natural Language Generation,NLG)任务是自然语言处理(Natural Languge Processing,NLP)任务中的一个子类,并且是一项具有挑战性的任务。随着深度学习在自然语言处理中的大量应用,其已经变成自然语言生成中处理各种任务的主要方法。自然语言生成任务中主要有问答任务、生成摘要任务、生成评论任务、机器翻译任务、生成式对话任务等。传统的生成模型依赖输入文本,基于有限的知识生成文本。为解决这个问题,引入了知识增强的方法。首先介绍了自然语言生成的研究背景和重要模型,然后针对自然语言处理归纳介绍了提高模型性能的方法,以及基于内部知识(如提取关键词增强生成、围绕主题词等)和外部知识(如借助外部知识图谱增强生成)集成到文本生成过程中的方法和架构。最后,通过分析生成任务面临的一些问题,讨论了未来的挑战和研究方向。

关键词: 自然语言生成, 知识增强, 深度学习, 知识图谱, 关键词提取, 主题词

Abstract: Natural language generation(NLG) task is a subclass of natural language processing(NLP) tasks and is a challenging task.With the massive application of deep learning in natural language processing,it has become the main method for handling various tasks in natural language generation.The main natural language generation tasks are question and answer tasks,summary generation tasks,comment generation tasks,machine translation tasks,generative dialogue tasks,etc.Traditional generative mo-dels rely on input text to generate text based on limited knowledge,and knowledge enhancement methods are introduced to solve this problem.Firstly,the research background and important models of natural language generation are introduced.Then,methods to improve model performance are introduced for natural language processing induction,and the methods and architectures based on the integration of internal knowledge(such as extracting keywords to enhance generation,surrounding subject words,etc.) and external knowledge(such as enhanced generation with the help of external knowledge graph) into the text generation process are introduced..Finally,the future challenges and research directions are discussed by analyzing some problems faced by the generation task.

Key words: Natural language generation, Knowledge enhancement, Deep learning, Knowledge graph, Keyword extraction, Subject headings

中图分类号: 

  • TP391
[1]LECUN Y,BENGIO Y,HINTON G.Deep learning[J].Nature,2015,521(7553):436-444.
[2]LIU J W,LIU Y,LUO X L.Research Progress in Deep Learning[J].Computer Application Research,2014,31(7):11.
[3]JI H,KE P,HUANG S,et al.Language generation with multi-hop reasoning on commonsense knowledge graph[C]//Procee-dings of the 2004 Conference on Empirical Methods in Natural Language Processing.2020:725-736.
[4]ZHOU H,YOUNG T,HUANG M,et al.Commonsense know-ledge aware conversation generation with graph attention[C]//IJCAI.2018:4623-4629.
[5]RADFORD A,WU J,CHILD R,et al.Language models are unsupervised multitask learners[J].OpenAI blog,2019,1(8):9.
[6]RAFFEL C,SHAZEER N,ROBERTS A,et al.Exploring thelimits of transfer learning with a unified text-to-text transfor-mer[J].Journal of Machine Learning Research,2020,140:1-140:67.
[7]YU W,ZHU C,LI Z,et al.A survey of knowledge-enhancedtext generation[J].arXiv:2010.04389,2020.
[8]GOODFELLOW I,BENGIO Y,COURVILLE A.Deep learning[M].MIT Press,2016.
[9]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[10]TENENBAUM J B,DE SILVA V,LANGFORD J C.A global geometric framework for nonlinear dimensionality reduction[J].Science,2000,290(5500):2319-2323.
[11]VINCENT P,LAROCHELLE H,BENGIO Y,et al.Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning.2008:1096-1103.
[12]DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:4171-4186.
[13]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Advances inNneural Information Processing Systems.2017:5998-6008.
[14]KINGMA D P,WELLING M.Auto-Encoding Variational Bayes[J].Stat,2014,1050:1.
[15]BAHDANAU D,CHO K H,BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2015.
[16]FU Z,SHI B,LAM W,et al.Partially-Aligned Data-to-TextGeneration with Distant Supervision[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP).2020:9183-9193.
[17]LUONG M T,PHAM H,MANNING C D.Effective Approaches to Attention-based Neural Machine Translation[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.2015:1412-1421.
[18]PUDUPPULLY R,DONG L,LAPATA M.Data-to-text generation with content selection and planning[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019:6908-6915.
[19]LIN Z,FENG M,SANTOS C N,et al.A structured self-attentive sentence embedding[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2017.
[20]ZHANG H,GOODFELLOW I,METAXAS D,et al.Self-attention generative adversarial networks[C]//International Conference on Machine Learning.PMLR,2019:7354-7363.
[21]LI W,XU J,HE Y,et al.Coherent Comments Generation for Chinese Articles with a Graph-to-Sequence Model[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:4843-4852.
[22]HAO J,WANG X,SHI S,et al.Multi-Granularity Self-Attention for Neural Machine Translation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:887-897.
[23]KIPF T N,WELLING M.Semi-supervised classification withgraph convolutional networks[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2017.
[24]MIHALCEA R,TARAU P.Textrank:Bringing order into text[C]//Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.2004:404-411.
[25]WU L,CHEN Y,SHEN K,et al.Graph Neural Networks forNatural Language Processing:A Survey[J].arXiv:2106.06090,2021.
[26]LIU Y,LIN Z,LIU F,et al.Generating paraphrase with topic as prior knowledge[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.2019:2381-2384.
[27]JIN K,ZHANG X,ZHANG J.Learning to Generate Diverse and Authentic Reviews via an Encoder-Decoder Model with Transformer and GRU[C]//2019 IEEE International Conference on Big Data(Big Data).IEEE,2019:3180-3189.
[28]NARAYAN S,COHEN S B,LAPATA M.Don’t Give Me the Details,Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:1797-1807.
[29]YANG P,LI L,LUO F,et al.Enhancing topic-to-essay genera-tion with external commonsense knowledge[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:2002-2012.
[30]XING C,WU W,WU Y,et al.Topic aware neural response gene-ration[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2017.
[31]MOU L,SONG Y,YAN R,et al.Sequence to Backward andForward Sequences:A Content-Introducing Approach to Genera-tive Short-Text Conversation[C]//The 26th International Conference on Computational Linguistics:Technical Papers(COLING 2016).2016:3349-3358.
[32]SERBAN I,KLINGER T,TESAURO G,et al.Multiresolution recurrent neural networks:An application to dialogue response generation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2017.
[33]LI H,ZHU J,ZHANG J,et al.Keywords-guided abstractivesentence summarization[C]//Proceedings of the AAAI Confe-rence on Artificial Intelligence.2020:8196-8203.
[34]BORDES A,USUNIER N,GARCIA-DURAN A,et al.Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS).2013:1-9.
[35]ZHANG H,LIU Z,XIONG C,et al.Grounded ConversationGeneration as Guided Traverses in Commonsense Knowledge Graphs[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:2031-2043.
[36]ZHANG Y,DAI H,KOZAREVA Z,et al.Variational Reasoning for Question Answering With Knowledge Graph[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2018.
[37]SUN H,DHINGRA B,ZAHEER M,et al.Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:4231-4242.
[38]SUN H,BEDRAX-WEISS T,COHEN W.PullNet:Open Do-main Question Answering with Iterative Retrieval on Know-ledge Bases and Text[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2380-2390.
[39]SAXENA A,TRIPATHI A,TALUKDAR P.Improving multi-hop question answering over knowledge graphs using knowledge base embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:4498-4507.
[40]LUO F,DAI D,YANG P,et al.Learning to control the fine-grained sentiment for story ending generation[C]//Proceedings of the 57th Annual Meeting of the Association for ComputationalLinguistics.2019:6020-6026.
[41]QIAO L,YAN J,MENG F,et al.A Sentiment-ControllableTopic-to-Essay Generator with Topic Knowledge Graph[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:Findings.2020:3336-3344.
[42]PAPINENI K,ROUKOS S,WARD T,et al.Bleu:a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.2002:311-318.
[43]DODDINGTON G.Automatic evaluation of machine translationquality using n-gram co-occurrence statistics[C]//Proceedings of the Second International Conference on Human Language Technology Research.2002:138-145.
[44]LAVIE A,AGARWAL A.METEOR:An automatic metric for MT evaluation with high levels of correlation with human judgments[C]//Proceedings of the Second Workshop on Statistical Machine Translation.2007:228-231.
[45]SERBAN I V,SORDONI A,BENGIOY,et al.Hierarchical neural network generative models for movie dialogues[J].arXiv:1507.04808,2015.
[46]RUS V,LINTEAN M.An optimal assessment of natural language student input using word-to-word similarity metrics[C]//International Conference on Intelligent Tutoring Systems.Berlin:Springer,2012:675-676.
[47]WIETING J,BANSAL M,GIMPEL K,et al.Towards universalparaphrastic sentence embeddings[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR 2016).2016.
[48]FORGUES G,PINEAU J,LARCHEVÊQUE J M,et al.Boot-strapping dialog systems with word embeddings[C]//Nips,Modern Machine Learning and Natural Language Processing Workshop.2014.
[49]XU P,HU Q.An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Long Papers).2018:1448-1457.
[50]ISO H,UEHARA Y,ISHIGAKI T,et al.Learning to select,track,and generate for data-to-text[J].Journal of Natural Language Processing,2020,27(3):599-626.
[51]TRISEDYA B,QI J,ZHANG R.Sentence generation for entity description with content-plan attention[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:9057-9064.
[52]SHAHIDI H,LI M,LIN J.Two Birds,One Stone:A Simple,Unified Model for Text Generation from Structured and Unstructured Data[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:3864-3870.
[53]GU J,LU Z,LI H,et al.Incorporating Copying Mechanism in Sequence-to-Sequence Learning[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(Long Papers).2016:1631-1640.
[54]SEE A,LIU P J,MANNING C D.Get To The Point:Summarization with Pointer-Generator Networks[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers).2017:1073-1083.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发