知识增强的自然语言生成研究综述

doi:10.11896/jsjkx.220200120

Abstract

Abstract: Natural language generation(NLG) task is a subclass of natural language processing(NLP) tasks and is a challenging task.With the massive application of deep learning in natural language processing,it has become the main method for handling various tasks in natural language generation.The main natural language generation tasks are question and answer tasks,summary generation tasks,comment generation tasks,machine translation tasks,generative dialogue tasks,etc.Traditional generative mo-dels rely on input text to generate text based on limited knowledge,and knowledge enhancement methods are introduced to solve this problem.Firstly,the research background and important models of natural language generation are introduced.Then,methods to improve model performance are introduced for natural language processing induction,and the methods and architectures based on the integration of internal knowledge(such as extracting keywords to enhance generation,surrounding subject words,etc.) and external knowledge(such as enhanced generation with the help of external knowledge graph) into the text generation process are introduced..Finally,the future challenges and research directions are discussed by analyzing some problems faced by the generation task.

Key words: Natural language generation, Knowledge enhancement, Deep learning, Knowledge graph, Keyword extraction, Subject headings

CLC Number:

TP391

LIANG Mingxuan, WANG Shi, ZHU Junwu, LI Yang, GAO Xiang, JIAO Zhixiang. Survey of Knowledge-enhanced Natural Language Generation Research[J].Computer Science, 2023, 50(6A): 220200120-8.

References

[1]LECUN Y,BENGIO Y,HINTON G.Deep learning[J].Nature,2015,521(7553):436-444.
[2]LIU J W,LIU Y,LUO X L.Research Progress in Deep Learning[J].Computer Application Research,2014,31(7):11.
[3]JI H,KE P,HUANG S,et al.Language generation with multi-hop reasoning on commonsense knowledge graph[C]//Procee-dings of the 2004 Conference on Empirical Methods in Natural Language Processing.2020:725-736.
[4]ZHOU H,YOUNG T,HUANG M,et al.Commonsense know-ledge aware conversation generation with graph attention[C]//IJCAI.2018:4623-4629.
[5]RADFORD A,WU J,CHILD R,et al.Language models are unsupervised multitask learners[J].OpenAI blog,2019,1(8):9.
[6]RAFFEL C,SHAZEER N,ROBERTS A,et al.Exploring thelimits of transfer learning with a unified text-to-text transfor-mer[J].Journal of Machine Learning Research,2020,140:1-140:67.
[7]YU W,ZHU C,LI Z,et al.A survey of knowledge-enhancedtext generation[J].arXiv:2010.04389,2020.
[8]GOODFELLOW I,BENGIO Y,COURVILLE A.Deep learning[M].MIT Press,2016.
[9]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[10]TENENBAUM J B,DE SILVA V,LANGFORD J C.A global geometric framework for nonlinear dimensionality reduction[J].Science,2000,290(5500):2319-2323.
[11]VINCENT P,LAROCHELLE H,BENGIO Y,et al.Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning.2008:1096-1103.
[12]DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:4171-4186.
[13]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Advances inNneural Information Processing Systems.2017:5998-6008.
[14]KINGMA D P,WELLING M.Auto-Encoding Variational Bayes[J].Stat,2014,1050:1.
[15]BAHDANAU D,CHO K H,BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2015.
[16]FU Z,SHI B,LAM W,et al.Partially-Aligned Data-to-TextGeneration with Distant Supervision[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP).2020:9183-9193.
[17]LUONG M T,PHAM H,MANNING C D.Effective Approaches to Attention-based Neural Machine Translation[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.2015:1412-1421.
[18]PUDUPPULLY R,DONG L,LAPATA M.Data-to-text generation with content selection and planning[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019:6908-6915.
[19]LIN Z,FENG M,SANTOS C N,et al.A structured self-attentive sentence embedding[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2017.
[20]ZHANG H,GOODFELLOW I,METAXAS D,et al.Self-attention generative adversarial networks[C]//International Conference on Machine Learning.PMLR,2019:7354-7363.
[21]LI W,XU J,HE Y,et al.Coherent Comments Generation for Chinese Articles with a Graph-to-Sequence Model[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:4843-4852.
[22]HAO J,WANG X,SHI S,et al.Multi-Granularity Self-Attention for Neural Machine Translation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:887-897.
[23]KIPF T N,WELLING M.Semi-supervised classification withgraph convolutional networks[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR).2017.
[24]MIHALCEA R,TARAU P.Textrank:Bringing order into text[C]//Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.2004:404-411.
[25]WU L,CHEN Y,SHEN K,et al.Graph Neural Networks forNatural Language Processing:A Survey[J].arXiv:2106.06090,2021.
[26]LIU Y,LIN Z,LIU F,et al.Generating paraphrase with topic as prior knowledge[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.2019:2381-2384.
[27]JIN K,ZHANG X,ZHANG J.Learning to Generate Diverse and Authentic Reviews via an Encoder-Decoder Model with Transformer and GRU[C]//2019 IEEE International Conference on Big Data(Big Data).IEEE,2019:3180-3189.
[28]NARAYAN S,COHEN S B,LAPATA M.Don’t Give Me the Details,Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:1797-1807.
[29]YANG P,LI L,LUO F,et al.Enhancing topic-to-essay genera-tion with external commonsense knowledge[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:2002-2012.
[30]XING C,WU W,WU Y,et al.Topic aware neural response gene-ration[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2017.
[31]MOU L,SONG Y,YAN R,et al.Sequence to Backward andForward Sequences:A Content-Introducing Approach to Genera-tive Short-Text Conversation[C]//The 26th International Conference on Computational Linguistics:Technical Papers(COLING 2016).2016:3349-3358.
[32]SERBAN I,KLINGER T,TESAURO G,et al.Multiresolution recurrent neural networks:An application to dialogue response generation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2017.
[33]LI H,ZHU J,ZHANG J,et al.Keywords-guided abstractivesentence summarization[C]//Proceedings of the AAAI Confe-rence on Artificial Intelligence.2020:8196-8203.
[34]BORDES A,USUNIER N,GARCIA-DURAN A,et al.Translating Embeddings for Modeling Multi-relational Data[C]//Neural Information Processing Systems(NIPS).2013:1-9.
[35]ZHANG H,LIU Z,XIONG C,et al.Grounded ConversationGeneration as Guided Traverses in Commonsense Knowledge Graphs[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:2031-2043.
[36]ZHANG Y,DAI H,KOZAREVA Z,et al.Variational Reasoning for Question Answering With Knowledge Graph[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2018.
[37]SUN H,DHINGRA B,ZAHEER M,et al.Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:4231-4242.
[38]SUN H,BEDRAX-WEISS T,COHEN W.PullNet:Open Do-main Question Answering with Iterative Retrieval on Know-ledge Bases and Text[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2380-2390.
[39]SAXENA A,TRIPATHI A,TALUKDAR P.Improving multi-hop question answering over knowledge graphs using knowledge base embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:4498-4507.
[40]LUO F,DAI D,YANG P,et al.Learning to control the fine-grained sentiment for story ending generation[C]//Proceedings of the 57th Annual Meeting of the Association for ComputationalLinguistics.2019:6020-6026.
[41]QIAO L,YAN J,MENG F,et al.A Sentiment-ControllableTopic-to-Essay Generator with Topic Knowledge Graph[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:Findings.2020:3336-3344.
[42]PAPINENI K,ROUKOS S,WARD T,et al.Bleu:a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.2002:311-318.
[43]DODDINGTON G.Automatic evaluation of machine translationquality using n-gram co-occurrence statistics[C]//Proceedings of the Second International Conference on Human Language Technology Research.2002:138-145.
[44]LAVIE A,AGARWAL A.METEOR:An automatic metric for MT evaluation with high levels of correlation with human judgments[C]//Proceedings of the Second Workshop on Statistical Machine Translation.2007:228-231.
[45]SERBAN I V,SORDONI A,BENGIOY,et al.Hierarchical neural network generative models for movie dialogues[J].arXiv:1507.04808,2015.
[46]RUS V,LINTEAN M.An optimal assessment of natural language student input using word-to-word similarity metrics[C]//International Conference on Intelligent Tutoring Systems.Berlin:Springer,2012:675-676.
[47]WIETING J,BANSAL M,GIMPEL K,et al.Towards universalparaphrastic sentence embeddings[C]//Proceedings of 3rd International Conference on Learning Representations(ICLR 2016).2016.
[48]FORGUES G,PINEAU J,LARCHEVÊQUE J M,et al.Boot-strapping dialog systems with word embeddings[C]//Nips,Modern Machine Learning and Natural Language Processing Workshop.2014.
[49]XU P,HU Q.An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Long Papers).2018:1448-1457.
[50]ISO H,UEHARA Y,ISHIGAKI T,et al.Learning to select,track,and generate for data-to-text[J].Journal of Natural Language Processing,2020,27(3):599-626.
[51]TRISEDYA B,QI J,ZHANG R.Sentence generation for entity description with content-plan attention[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:9057-9064.
[52]SHAHIDI H,LI M,LIN J.Two Birds,One Stone:A Simple,Unified Model for Text Generation from Structured and Unstructured Data[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:3864-3870.
[53]GU J,LU Z,LI H,et al.Incorporating Copying Mechanism in Sequence-to-Sequence Learning[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(Long Papers).2016:1631-1640.
[54]SEE A,LIU P J,MANNING C D.Get To The Point:Summarization with Pointer-Generator Networks[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers).2017:1073-1083.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Survey of Knowledge-enhanced Natural Language Generation Research

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 15

Metrics

Comments

Recommended 0

[1]	ZHANG Yian, YANG Ying, REN Gang, WANG Gang. Study on Multimodal Online Reviews Helpfulness Prediction Based on Attention Mechanism [J]. Computer Science, 2023, 50(8): 37-44.
[2]	SONG Xinyang, YAN Zhiyuan, SUN Muyi, DAI Linlin, LI Qi, SUN Zhenan. Review of Talking Face Generation [J]. Computer Science, 2023, 50(8): 68-78.
[3]	WANG Xu, WU Yanxia, ZHANG Xue, HONG Ruize, LI Guangsheng. Survey of Rotating Object Detection Research in Computer Vision [J]. Computer Science, 2023, 50(8): 79-92.
[4]	ZHOU Ziyi, XIONG Hailing. Image Captioning Optimization Strategy Based on Deep Learning [J]. Computer Science, 2023, 50(8): 99-110.
[5]	ZHANG Xiao, DONG Hongbin. Lightweight Multi-view Stereo Integrating Coarse Cost Volume and Bilateral Grid [J]. Computer Science, 2023, 50(8): 125-132.
[6]	TANG Shaosai, SHEN Derong, KOU Yue, NIE Tiezheng. Link Prediction Model on Temporal Knowledge Graph Based on Bidirectionally Aggregating Neighborhoods and Global Aware [J]. Computer Science, 2023, 50(8): 177-183.
[7]	WANG Yu, WANG Zuchao, PAN Rui. Survey of DGA Domain Name Detection Based on Character Feature [J]. Computer Science, 2023, 50(8): 251-259.
[8]	WANG Mingxia, XIONG Yun. Disease Diagnosis Prediction Algorithm Based on Contrastive Learning [J]. Computer Science, 2023, 50(7): 46-52.
[9]	SHEN Zhehui, WANG Kailai, KONG Xiangjie. Exploring Station Spatio-Temporal Mobility Pattern:A Short and Long-term Traffic Prediction Framework [J]. Computer Science, 2023, 50(7): 98-106.
[10]	HUO Weile, JING Tao, REN Shuang. Review of 3D Object Detection for Autonomous Driving [J]. Computer Science, 2023, 50(7): 107-118.
[11]	ZHOU Bo, JIANG Peifeng, DUAN Chang, LUO Yuetong. Study on Single Background Object Detection Oriented Improved-RetinaNet Model and Its Application [J]. Computer Science, 2023, 50(7): 137-142.
[12]	MAO Huihui, ZHAO Xiaole, DU Shengdong, TENG Fei, LI Tianrui. Short-term Subway Passenger Flow Forecasting Based on Graphical Embedding of Temporal Knowledge [J]. Computer Science, 2023, 50(7): 213-220.
[13]	LI Yuqiang, LI Linfeng, ZHU Hao, HOU Mengshu. Deep Learning-based Algorithm for Active IPv6 Address Prediction [J]. Computer Science, 2023, 50(7): 261-269.
[14]	LI Kun, GUO Wei, ZHANG Fan, DU Jiayu, YANG Meiyue. Adversarial Malware Generation Method Based on Genetic Algorithm [J]. Computer Science, 2023, 50(7): 325-331.
[15]	GAO Xiang, TANG Jiqiang, ZHU Junwu, LIANG Mingxuan, LI Yang. Study on Named Entity Recognition Method Based on Knowledge Graph Enhancement [J]. Computer Science, 2023, 50(6A): 220700153-6.