基于图卷积神经网络的文本分类方法研究综述

doi:10.11896/jsjkx.210800064

摘要/Abstract

摘要： 文本分类是自然语言处理领域中常见的任务,机器学习和深度学习在该任务中已有较多研究并取得了很大进展,然而,这些传统方法只能处理欧氏空间的数据,不能完全有效地表达出文本的语义信息。为了打破传统的学习模式,诸多研究开始尝试用图表示文本中各实体间的丰富关系,并利用图卷积神经网络学习文本表示。文中对基于图卷积神经网络的文本分类方法进行了综述,首先概述了图卷积神经网络的背景与原理;其次,利用不同类型的图网络详细阐述了基于图卷积神经网络的文本分类方法,同时分析了图卷积神经网络在网络深度上的局限性,并介绍了深层网络在文本分类任务上的最新进展;最后,通过实验比较了各模型的分类性能,并探讨了该领域的难点与未来的发展方向。

关键词: 非欧空间, 过平滑, 图卷积神经网络, 图注意力网络, 文本分类

Abstract: Text classification is a common task in natural language processing,in which there are a lot of research and progress based on machine learning and deep learning.However,these traditional methods can only process Euclidean spatial data,and cannot express the semantic information of the document effectively.To break the traditional learning mode,many recent studies start to use graphs to represent complicated relationships among entities in the document,and explore graph convolutional neural network for text representation.This paper reviews the text classification methods based on graph convolutional network.Firstly,the background and principle of graph convolutional network are summarized.Then,text classification methods based on graph convolutional network are described in detail according to different types of graph-based networks.Meanwhile,it analyzes the limi-tation of graph convolutional network in the depth of networks,and introduces the latest developments of deep networks in text classification.Finally,the classification performance of models involved in this paper is compared through some experiments,and the challenges and future research direction in this field are discussed.

Key words: Graph attention network, Graph convolutional network, Non-Euclidean space, Over-smoothing, Text classification

中图分类号:

TP391.1

檀莹莹, 王俊丽, 张超波. 基于图卷积神经网络的文本分类方法研究综述[J]. 计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064

TAN Ying-ying, WANG Jun-li, ZHANG Chao-bo. Review of Text Classification Methods Based on Graph Convolutional Network[J]. Computer Science, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064

参考文献

[1]WANG J,WANG Z,ZHANG D,et al.Combining knowledge with deep convolutional neural networks for short text classification[C]//Proceedings of the 26th International Joint Confe-rence on Artificial Intelligence.2017:2915-2921.
[2]HU X J,LIU L,QIU N J.A Novel Spam Categorization Algorithm Based on Active Learning Method and Negative Selection Algorithm[J].Dian Zi Xue Bao/Acta Electronica Sinica,2018,46(1):203-209.
[3]MAAS A,DALY R E,PHAM P T,et al.Learning word vectors for sentiment analysis[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies.2011:142-150.
[4]SU J S,ZHANG B F,XU X.Advances in Machine Learning Based Text Categorization[J].Ruan Jian Xue Bao/Journal of Software,2006,17(9):1848-1859.
[5]WU Y J,LI J,SONG C F,et al.High Utility Neural Networks for Text Classification[J].Dian Zi Xue Bao:Acta Electronica Sinica,2020,48(2):279-284.
[6]COVER T,HART P.Nearest neighbor pattern classification[J].IEEE Transactions on Information Theory,1967,13(1):21-27.
[7]JOACHIMS T.Text categorization with support vector ma-chines:Learning with many relevant features[C]//European Conference on Machine Learning.Berlin:Springer,1998:137-142.
[8]WANG S I,MANNING C D.Baselines and bigrams:Simple,good sentiment and topic classification[C]//Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics.2012:90-94.
[9]KALCHBRENNER N,GREFENSTETTE E,BLUNSOM P.A convolutional neural network for modelling sentences[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.2014:655-665.
[10]ZHU X,SOBHANI P,GUO H.Long short-term memory over recursive structures[C]//Proceedings of the 32ndInternational Conference on Machine Learning.2015:1604-1612.
[11]KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.2014:1746-1751.
[12]ZHANG X,ZHAO J,LECUN Y.Character-level convolutional networks for text classification[C]//Advances in Neural Information Processing Systems.2015:649-657.
[13]LIU P,QIU X,HUANG X.Recurrent Neural Network for Text Classification with Multi-Task Learning[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence.2016:2873-2879.
[14]YANG Z,YANG D,DYER C,et al.Hierarchical Attention Networks for Document Classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.2016:1480-1489.
[15]BAHDANAU D,CHO K,BENGIO Y.Neural Machine Translation by Jointly Learning to Align and Translate[J].arXiv:1409.0473,2014.
[16]XU B B,CEN K T,HUANG J J,et al.A Survey on Graph Convolutional Neural Network[J].Ji Suan Ji Xue Bao/Chinese Journal of Computers,2020,43(5):755-780.
[17]WU Z,PAN S,CHEN F,et al.A Comprehensive Survey onGraph Neural Networks[J].IEEE Transactions on Neural Networks and Learning Systems,2021,32(1):4-24.
[18]ZHANG Z,CUI P,ZHU W.Deep Learning on Graphs:A Survey[J].IEEE Transactions on Knowledge and Data Enginee-ring,2022,34(1):249-270.
[19]ZHOU J,CUI G,HU S,et al.Graph neural networks:A review of methods and applications[J].AI Open,2020,1:57-81.
[20]MINAEE S,KALCHBRENNER N,CAMBRIA E,et al.DeepLearning Based Text Classification:A Comprehensive Review[J].arXiv:2004.03705v1,2020.
[21]LI Q,PENG H,LI J,et al.A Survey on Text Classification:From Shallow to Deep Learning[J].arXiv:2008.00364,2020.
[22]KIPF T N,WELLING M.A PyTorch implementation of GCN[EB/OL].https://github.com/tkipf/pygcn.
[23]HAMILTON W,YING Z,LESKOVEC J.A PyTorch imple-mentation of GraphSage[EB/OL].https://github.com/williamleif/graphsage-simple/.
[24]VELIKOVI P,CUCURULL G,CASANOVA A,et al.A Tensorflow implementation of GAT[EB/OL].https://github.com/PetarV-/GAT.
[25]CHEN J,MA T,XIAO C.A Tensorflow implementation ofFastGCN[EB/OL].https://github.com/matenure/FastGCN.
[26]WANG M,ZHENG D,YE Z,et al.A PyTorch implementation of JKNet [EB/OL].https://github.com/dmlc/dgl/tree/master/examples/pytorch/jknet.
[27]WU F,ZHANG T,SOUZA A,et al.A PyTorch implementation of SGC[EB/OL].https://github.com/Tiiiger/SGC.
[28]YAO L,MAO C,LUO Y.A Tensorflow implementation of Text-gcn[EB/OL].https://github.com/yao8839836/text_gcn.
[29]GAO H,CHEN Y,JI S.A Tensorflow implementation ofhConv-gPool-Net[EB/OL].https://github.com/HongyangGao/hConv-gPool-Net.
[30]KLICPERA J,BOJCHEVSKI A,GUNNEMANN S.Tensor-Flow and PyTorch implementations of PPNP and APPNP[EB/OL].https://github.com/klicperajo/ppnp.
[31]HUANG L,MA D,LI S,et al.A PyTorch implementation of Text-level GCN[EB/OL].https://github.com/mojave-pku/TextLevelGCN.
[32]HU L,YANG T,SHI C,et al.A PyTorch implementation ofHAGT[EB/OL].https://github.com/ytc272098215/HGAT.
[33]LIU X,YOU X,ZHANG X,et al.A Tensorflow implementation of TensorGCN[EB/OL].https://github.com/THUMLP/TensorGCN.
[34]RONG Y,HUANG W,XU T,et al.A PyTorch implementation of DropEdge[EB/OL].https://github.com/DropEdge/Drop-Edge.
[35]ZHANG J,MENG L.A PyTorch implementation of GResNet[EB/OL].https://github.com/jwzhanggy/GResNet.
[36]ZHAO L,AKOGLU L.A PyTorch implementation of PairNorm[EB/OL].https://github.com/LingxiaoShawn/PairNorm.
[37]CHEN G,TIAN Y,SONG Y.A PyTorch implementation ofD-GCN[EB/OL].https://github.com/cuhksz-nlp/DGSA.
[38]DING K,WANG J,LI J,et al.A PyTorch implementation of HyperGAT[EB/OL].https://github.com/kaize0409/HyperGAT_TextClassification.
[39]ZHANG Y,YU X,CUI Z,et al.A Tensorflow implementation of TextING[EB/OL].https://github.com/CRIPAC-DIG/Text-ING.
[40]YUAN L,WANG J,YU L,et al.A PyTorch implementation of G-ATT[EB/OL].https://github.com/YuanLi95/GATT-For-Aspect.
[41]FENG W,ZHANG J,DONG Y,et al.A PyTorch implementation of GRAND[EB/OL].https://github.com/THUDM/GRAND.
[42]ZHOU K,HUANG X,LI Y,et al.A PyTorch implementation of DGN[EB/OL].https://github.com/Kaixiong-Zhou/DGN.
[43]CHEN M,WEI Z,HUANG Z,et al.A PyTorch implementation of GCNII[EB/OL].https://github.com/chennnM/GCNII.
[44]BRUNA J,ZAREMBA W,SZLAM A,et al.Spectral networks and locally connected networks on graphs[J].arXiv:1312.6203,2013.
[45]CHUNG F R K.Spectral Graph Theory[EB/OL].http://www.ams.org/books/cbms/092/.
[46]SANDRYHAILA A,MOURA J M F.Discrete Signal Proces-sing on Graphs[J].IEEE Trans.on Signal Processing,2014,62(12):3042-3054.
[47]SHUMAN DI,NARANG S K,FROSSARD P,et al.The Emerging Field of Signal Processing on Graphs:Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains[J].IEEE Signal Processing Magazine,2013,30(3):83-98.
[48]DEFFERRARD M,BRESSON X,VANDERGHEYNST P.Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering[C]//Advances in Neural Information Processing Systems.2016:3844-3852.
[49]HAMMOND D K,VANDERGHEYNST P,GRIBONVAL R.Wavelets on graphs via spectral graph theory[J].Applied and Computational Harmonic Analysis,2011,30(2):129-150.
[50]KIPF T N,WELLING M.Semi-Supervised Classification withGraph Convolutional Networks[C]//Proceedings of the 5th International Conference on Learning Representations.2017:1-13.
[51]LI Y,TARLOW D,BROCKSCHMIDT M,et al.Gated graph sequence neural networks[J].arXiv:1511.05493,2016.
[52]YUN S,JEONG M,KIM R,et al.Graph Transformer Networks[C]//Advances in Neural Information Processing Systems.2019:11960-11970.
[53]YAO L,MAO C,LUO Y.Graph Convolutional Networks for Text Classification[C]//Proceedings of the 33rd AAAI Confe-rence on Artificial Intelligence.2019:7370-7377.
[54]CHURCH K W,HANKS P.Word association norms,mutual information,and lexicography[J].Computational linguistics,1990,16(1):22-29.
[55]ZAANEN M,KANTERS P.Automatic Mood Classificationusing TF*IDF based on Lyrics[C]//Proceedings of the 11th International Society for Music Information Retrieval Confe-rence.2010:75-80.
[56]LIU X,YOU X,ZHANG X,et al.Tensor graph convolutional networks for text classification[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence.2020:8409-8416.
[57]WU F,ZHANG T,SOUZA A,et al.Simplifying graph convolutional networks[C]//Proceedings of 36th International Confe-rence on Machine Learning.2019:6861-6871.
[58]HUANG L,MA D,LI S,et al.Text Level Graph Neural Network for Text Classification[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.2019:3442-3448.
[59]GILMER J,SCHOENHOLZ S S,RILEY P F,et al.Neural message passing for quantum chemistry[C]//Proceedings of the 34th International Conference on Machine Learning.2017:1263-1272.
[60]GAO H,CHEN Y,JI S.Learning graph pooling and hybrid convolutional operations for text representations[C]//Proceedings of the World Wide Web Conference.2019:2743-2749.
[61]VELIKOVI P,CUCURULL G,CASANOVA A,et al.GraphAttention Networks[J].arXiv:1710.10903,2018.
[62]HU L,YANG T,SHI C,et al.Heterogeneous Graph AttentionNetworks for Semi-supervised Short Text Classification[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing.2019:4821-4830.
[63]YUAN L,WANG J,YU L,et al.Graph Attention Networkwith Memory Fusion for Aspect-level Sentiment Analysis[C]//Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.2020:27-36.
[64]CHEN G,TIAN Y,SONG Y.Joint Aspect Extraction and Sentiment Analysis with Directional Graph Convolutional Networks[C]//Proceedings of the 28th International Conference on Computational Linguistics.2020:272-279.
[65]DING K,WANG J,LI J,et al.Be More with Less:Hypergraph Attention Networks for Inductive Text Classification[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.2020:4927-4936.
[66]BLEI D M,NG A Y,JORDAN M I.Latent dirichlet allocation[J].Journal of Machine Learning Research,2003,3:993-1022.
[67]ZHANG Y,YU X,CUI Z,et al.Every Document Owns ItsStructure:Inductive Text Classification via Graph Neural Networks[C]//Proceedings of the Annual Meeting of the Association for Computational Linguistics.2020:334-339.
[68]YUE Z,QI L,SONG L.Sentence-State LSTM for Text Representation[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.2018:317-327.
[69]LI W,LI S,MA S,et al.Recursive Graphical Neural Networks for Text Classification[J].arXiv:1909.08166,2020.
[70]HUANG B,CARLEY K M.Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.2019:5469-5477.
[71]ZHANG H,ZHANG J.Text Graph Transformer for Document Classification[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.2020:8322-8327.
[72]LI Q,HAN Z,WU X.Deeper insights into graph convolutional networks for semi-supervised learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence.2018:3538-3545.
[73]KLICPERA J,BOJCHEVSKI A,GUNNEMANN S.PredictThen Propagate:Graph Neural Networks Meet Personalized Pagerank[J].arXiv:1810.05997,2019.
[74]PAGE L,BRIN S,MOTWANI R,et al.The pagerank citation ranking:Bringing order to the web[EB/OL].http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf.
[75]RONG Y,HUANG W,XU T,et al.DropEdge:Towards Deep Graph Convolutional Networks on Node Classification[C]//Proceedings of the International Conference on Learning Representations.2020:1-17.
[76]FENG W,ZHANG J,DONG Y,et al.Graph Random NeuralNetwork for Semi-Supervised Learning on Graphs[C]//Advances in Neural Information Processing Systems.2020:22092-22103.
[77]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Computer Socie-ty Conference on Computer Vision and Pattern Recognition.2016:770-778.
[78]XU K,LI C,TIAN Y,et al.Representation Learning on Graphs with Jumping Knowledge Networks[C]//Proceedings of the 35th International Conference on Machine Learning.2018:5449-5458.
[79]ZHANG J,MENG L.GResNet:GraphResidual Network for Reviving Deep GNNs from Suspended Animation[J].arXiv:1909.05729,2020.
[80]CHEN M,WEI Z,HUANG Z,et al.Simple and Deep Graph Convolutional Networks[C]//Proceedings of 37th International Conference on Machine Learning.2020:1725-1735.
[81]HEN D,LIN Y,LI W,et al.Measuring and relieving the over-smoothing problem for graph neural networks from the topolo-gical view[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence.2020:3438-3445.
[82]DWIVEDI V,JOSHI C,LAURENT T,et al.Benchmarkinggraph neural networks[J].arXiv:2003.00982,2020.
[83]ZHAO L,AKOGLU L.Pairnorm:Tackling oversmoothing inGNNS[J].arXiv:1909.12223v1,2020.
[84]ZHOU K,HUANG X,LI Y,et al.Towards Deeper Graph Neural Networks with Differentiable Group Normalization[C]//Advances in Neural Information Processing Systems Conference.2020:1-12.
[85]SEN P,NAMATA G,BILGIC M,et al.Collective classification in network data[J].AI magazine,2008,29(3):93-106.
[86]HAMILTON W,YING Z,LESKOVEC J.Inductive representation learning on large graphs[C]//Advances in Neural Information Processing Systems.2017:1024-1034.
[87]CHEN J,MA T,XIAO C.FastGCN:fast learning with graph convolutional networks via importance sampling[J].arXiv:1801.10247,2018.
[88]YANG Z,COHEN W,SALAKHUTDINOV R.Revisiting semi-supervised learning with graph embeddings[C]//Proceedings of the International Conference on Machine Learning.2016:40-48.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed