计算机科学 ›› 2022, Vol. 49 ›› Issue (4): 288-293.doi: 10.11896/jsjkx.211100016

• 人工智能 • 上一篇    下一篇


钟桂凤1, 庞雄文2, 隋栋3   

  1. 1 广州理工学院计算机科学与工程学院 广州 510540;
    2 华南师范大学计算机学院 广州 530631;
    3 北京建筑大学电气与信息工程学院 北京 102406
  • 收稿日期:2021-11-01 修回日期:2022-01-23 发布日期:2022-04-01
  • 通讯作者: 钟桂凤(109488818@qq.com)
  • 基金资助:
    国家自然科学青年基金(61702026); 2020年度广东省高校科研项目(2020GXJK201); 2019年度广东省高校科研项目(2019KTSCX243); 2021年广东省高等教育专项(2021GXJK275)

Text Classification Method Based on Word2Vec and AlexNet-2 with Improved AttentionMechanism

ZHONG Gui-feng1, PANG Xiong-wen2, SUI Dong3   

  1. 1 College of Computer Science&Engineering, Guangzhou Institute of Science and Technology, Guangzhou 510540, China;
    2 School of Computer, South China Normal University, Guangzhou 530631, China;
    3 School of Electrical and Information Engineering, Beijing University of Civil Engineering and Architecture, Beijing 102406, China
  • Received:2021-11-01 Revised:2022-01-23 Published:2022-04-01
  • About author:ZHONG Gui-feng,born in 1983,postgraduate,lecturer.Her main research interests include data analysis and mi-ning,machine learning and applications of artificial intelligence.
  • Supported by:
    This work was supported by the National Natural Science Youth Fund(61702026),2020 Guangdong University Scientific Research Project(2020GXJK201),2019 Guangdong University Scientific Research Project(2019KTSCX243) and Special Program for Higher Education in Guangdong Province in 2021(2021GXJK275).

摘要: 为了提高文本分类的准确性和运行效率,提出一种Word2Vec文本表征和改进注意力机制AlexNet-2的文本分类方法。首先,利用Word2Vec对文本词特征进行嵌入表示,并训练词向量,将文本表示成分布式向量的形式;然后,利用改进的AlexNet-2对长距离词相依性进行有效编码,同时对模型添加注意力机制,以高效学习目标词的上下文嵌入语义,并根据词向量的输入与最终预测结果的相关性,进行词权重的调整。实验在3个公开数据集中进行评估,分析了大量样本标注和少量样本标注的情形。实验结果表明,与已有的优秀方法相比,所提方法可以明显提高文本分类的性能和运行效率。

关键词: AlexNet-2模型, 词相依性, 上下文嵌入, 文本分类, 注意力机制

Abstract: In order to improve the accuracy and efficiency of text classification, a text classification method based on Word2Vec text representation and AlexNet-2 with improved attention mechanism is proposed.Firstly, Word2Vec is adopted to embed the text word features, and the word vector is trained to represent the text in the form of distributed vectors.Then, an improved AlexNet-2 is used to effectively encode the long-distance word dependency.Meanwhile, the attention mechanism is added to the model to learn the contextual embedding semantics of the target word efficiently, and the word weight is adjusted according to the correlation between the input of word vector and the final prediction result.The experiment is evaluated in three public data sets, and the situations of a large number of sample annotations and a small number of sample annotations are analyzed.Experimental results show that, compared with the existing excellent methods, the proposed method can significantly improve the performance and efficiency of text classification.

Key words: AlexNet-2 model, Attention mechanism, Contextual embedding, Text classification, Word dependency


  • TP391
[1] LUO X.Efficient English Text Classification Using SelectedMachine Learning Techniques[J].Alexandria Engineering Journal,2021,60(3):3401-3409.
[2] AL-SALEMI B,AYOB M,NOAH S.Feature Ranking for Enhancing Boosting-based Multi-label Text Categorization[J].Expert Systems with Applications,2018,113(12):531-543.
[3] JIN G,XU L.Long Text Classification Method,Device,Com-puter Equipment and Storage Medium Based on Word Bag Mo-del:CN Patent 110096591 A[P].2019.
[4] XU A D,ZHAO Y K,ZHANG Y Q,et al.Text Classification Method Based on Heterogeneous Space and Multiple-Classifier Fusion[J].Journal of Sichuan Ordnance,2019,40(12):136-141.
[5] AL-SALEMI B,AB-AZIZ M J,MOHD-NOAH S A.et al.LDA-AdaBoost.MH:Accelerated AdaBoost.MH Based on Latent Dirichlet Allocation for Text Categorization[J].Journal of Information Science,2015,41(1):27-40.
[6] SHI Z S,DU Y,DU T,et al.The Turnout Abnormality Diagnosis Based on Semi-Supervised Learning Method[J].International Journal of Software Engineering and Knowledge Engineering,2020,30(7):961-976.
[7] WU Y J,LI J,SONG C F,et al.High Utility Neural Networks for Text Classification[J].Acta Electronica Sinica,2020,48(2):279-284.
[8] DU L,CAO D,LIN S Y,et al.Extraction and Automatic Classi-fication of TCM Medical Records Based on Attention Mechanism of BERT and Bi-LSTM[J].Computer Science,2020,47(S2):426-430.
[9] YAO L,MAO C S,LUO Y.Graph Convolutional Networks for Text Classification[J].arXiv:1809.05679,2018.
[10] RAGESH R,SELLAMANNICKAM S,IYER A,et al.HeteGCN:Heterogeneous Graph Convolutional Networks for Text Classification[C]//The Fourteenth ACM International Conference on Web Search and Data Mining.Virtual Event,Israel:ACM,2021:105-115.
[11] KIPF T N,WELLING M.Semi-Supervised Classification withGraph Convolutional Networks[C]//International Conference on Learning Representations.Toulon,France:IEEE,2017:109-117.
[12] XIAO L,CHEN B L,HUANG X,et al.Multi-label Text Classification Method Based on Label Semantic Information[J].Journal of Software,2020,31(4):1079-1089.
[13] JANG B,KIM M,HARERIMANA G,et al.Bi-LSTM Model to Increase Accuracy in Text Classification:Combining Word2vec CNN and Attention Mechanism[J].Applied Sciences,2020,10(17):5841-5750.
[14] LUO X,WANG X H.Research on Multi-feature Fusion Text Classification Model Based on Self-attention Mechanism[J].Journal of Physics:Conference Series,2020,1693(1):012071-012077.
[15] TU N,THU H,NGUYEN V A.Language Model Combinedwith Word2Vec for Product’s Aspect Based Extraction[J].ICIC Express Letters,2020,14(11):1033-1040.
[16] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Classification with Deep Convolutional Neural Networks[J].Advances in Neural Information Processing Systems,2012,25(2):1-9.
[17] KRIZHEVSKY A,SUTSKEVER I,GEOFFREY E.ImageNet Classification with Deep Convolutional Neural Networks[J].Communications of the ACM,2017,60(6):84-90.
[18] BAHDANAU D,CHO K,BENGIO Y.Neural Machine Translation by Jointly Learning to Align and Translate[J].Computer Science,2014,47(3):1-15.
[19] LIU J Y,JIA B B.Combining One-vs-One Decomposition andInstance-Based Learning for Multi-Class Classification[J].IEEE Access,2020,85(8):499-507.
[20] GAO C L,XU H,GAO K.Attention-based Bi-LSTM Network with Part-of-speech Features for Chinese Text Classification[J].Journal of Hebei University of Science and Technology,2018,39(5):73-80.
[21] WANG T,LI M.Research on Comment Text Mining Based on LDA Model and Semantic Network[J].Journal of Chongqing Technology and Business University(Natural Science Edition),2019,36(4):9-16.
[1] 周芳泉, 成卫青.
Sequence Recommendation Based on Global Enhanced Graph Neural Network
计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[2] 戴禹, 许林峰.
Cross-image Text Reading Method Based on Text Line Matching
计算机科学, 2022, 49(9): 139-145. https://doi.org/10.11896/jsjkx.220600032
[3] 周乐员, 张剑华, 袁甜甜, 陈胜勇.
Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion
计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[4] 熊丽琴, 曹雷, 赖俊, 陈希亮.
Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization
计算机科学, 2022, 49(9): 172-182. https://doi.org/10.11896/jsjkx.210800112
[5] 饶志双, 贾真, 张凡, 李天瑞.
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[6] 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥.
Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism
计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[7] 孙奇, 吉根林, 张杰.
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[8] 檀莹莹, 王俊丽, 张超波.
Review of Text Classification Methods Based on Graph Convolutional Network
计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[9] 闫佳丹, 贾彩燕.
Text Classification Method Based on Information Fusion of Dual-graph Neural Network
计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[10] 郝志荣, 陈龙, 黄嘉成.
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[11] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[12] 武红鑫, 韩萌, 陈志强, 张喜龙, 李慕航.
Survey of Multi-label Classification Based on Supervised and Semi-supervised Learning
计算机科学, 2022, 49(8): 12-25. https://doi.org/10.11896/jsjkx.210700111
[13] 汪鸣, 彭舰, 黄飞虎.
Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction
计算机科学, 2022, 49(8): 40-48. https://doi.org/10.11896/jsjkx.220100188
[14] 金方焱, 王秀利.
Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM
计算机科学, 2022, 49(7): 179-186. https://doi.org/10.11896/jsjkx.210500190
[15] 熊罗庚, 郑尚, 邹海涛, 于化龙, 高尚.
Software Self-admitted Technical Debt Identification with Bidirectional Gate Recurrent Unit and Attention Mechanism
计算机科学, 2022, 49(7): 212-219. https://doi.org/10.11896/jsjkx.210500075
Full text



No Suggested Reading articles found!