Computer Science ›› 2020, Vol. 47 ›› Issue (11A): 508-514.doi: 10.11896/jsjkx.191100041

• Big Data & Data Science • Previous Articles     Next Articles

Multimodal Sentiment Analysis Based on Attention Neural Network

LIN Min-hong, MENG Zu-qiang   

  1. School of Computer,Electronics and Information,Guangxi University,Nanning 530004,China
  • Online:2020-11-15 Published:2020-11-17
  • About author:LIN Min-hong,born in 1995,postgraduate.Her main research interests include data mining and cross-media mining.
    MENG Zu-qiang,born in 1974,Ph.D,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include data mining,cross-media mining and KDD (Knowledge Discovery in Database).
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61762009).

Abstract: In recent years,more and more people are keen to express their feelings and opinions in the form of both pictures and texts on social media,and the scale of multimodal data including images and texts keeps growing.Compared with single mode data,multimodal data contains more information.It can better reveal the real emotion of users.Sentiment analysis of these huge amounts of multimodal data helps to better understand people's attitudes and opinions.In addition,it has a wide range of applications.In order to solve the problem of information redundancy in multimodal sentiment analysis task,this paper proposes a multimodal sentiment analysis method based on tensor fusion scheme and attention neural network.This method constructs the text feature extraction model and image feature extraction model based on attention neural network to highlight the key areas of image emotion information and words containing emotion information,so as to make the expression of each feature more concise and accurate.It fuses each modal feature using tensor fusion method in order to obtain the joint feature vector.Finally,it uses support vector machine for sentiment classification.The experimental results of this model on two real Twitter data sets show that compared with other sentiment analysis models,this method has a great improvement in precision rate,recall rate,F1 score andaccuracy rate.

Key words: Social media, Multimodal data, Sentiment analysis, Attention mechanism, Tensor fusion

CLC Number: 

  • TP391
[1] ASUR S,HUBERMAN B A.Predicting the future with socialmedia [C] //Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.Toronto,Canada,2010:492-499.
[2] O'CONNOR B,BALASUBRAMANYAN R,ROUTLEDGE B R,et al.From tweets to polls:Linking text sentiment to public opinion time series [C]//Proceedings of the International AAAI Conference on Weblogs And Social Media.Washington,United States,2010:11:122-129.
[3] TUMASJAN A,SPRENGER T O,SANDNER P G,et al.Predicting elections with twitter:What 140 characters reveal about political sentiment[C]//Proceedings of the International AAAI Conference on Weblogs And Social Media.Washington,USA,2010:10:122-129.
[4] WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional Block Attention Module[C]//Proceedings of the European Conference on Computer Vision 2018.ECCV,Munich,Germany,2018:3-19.
[5] LI X,XIE H,CHEN L,et al.News impact on stock price return via sentiment analysis[J].Knowledge-Based Systems,2014,69:14-23.
[6] NGUYEN T H,SHIRAI K.Topic modeling based sentimentanalysis on social media for stock market prediction[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).Beijing,China,2015:1354-1364.
[7] ZHANG L,ZHAO Y,ZHU Z F.Advances in SemanticallyShared Subspace Learning for Cross-Media Data[J].Chinese Journal of Computers,2017.
[8] ZHANG L,WANG S,LIU B.Deep learning for sentiment analysis:A survey[J].Wiley Interdisciplinary Reviews:Data Mining and Knowledge Discovery,2018:e1253.
[9] TURNEY P D.Thumbs up or thumbs down:semantic orientation applied to unsupervised classification of reviews[J].Proceedings of Annual Meeting of the Association for Computational Linguistics,2002:417-424.
[10] TABOADAM,BROOKE J,TOFILOSKI M,et al.Lexicon-Based Methods for Sentiment Analysis[J].Computational Linguistics,2011,37(2):267-307.
[11] BACCIANELLA S,ESULI A,SEBASTIANI F.Sentiwordnet3.0:an enhanced lexical resource for sentiment analysis and opinion mining[C]//Proceedings of the International Conference on Language Resources and Evaluation.Valletta,Malta,European.2010,2010(10):2200-2204.
[12] HU M,LIU B.Mining and summarizing customer reviews[C]//Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.ACM,2004:168-177.
[13] PANG B,LEE L,VAITHYANATHAN S.Thumbs up?:sentiment classification using machine learning techniques [C]//Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing.2002.
[14] KIM Y.Convolutional neural networks for sentence classification[J].arXiv,2014:1408.5882.
[15] KALCHBRENNER N,GREFENSTETTE E,BLUNSOM P.A Convolutional Neural Network for Modelling Sentences[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Association for Computational Linguistics,2014:655-665.
[16] TAY Y,TUAN L A,HUI S C.Learning to attend via word-as-pect associative fusion for aspect-based sentiment analysis[C]//Thirty-Second AAAI Conference on Artificial Intelligence,Louisiana,USA,2018.
[17] TANG D,QIN B,LIU T.Document modeling with gated re-current neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.Lisbon,Portugal,2015:1422-1432.
[18] SONG J,YU Q,SONG Y Z,et al.Deep spatial-semantic attention for fine-grained sketch-based image retrieval[C]//Proceedings of the IEEE International Conference on Computer Vision(ICCV 2017).Venice,Italy,2017:5551-5560.
[19] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[20] SIERSDORFER S,MINACK E,DENG F,et al.Analyzing and predicting sentiment of images on the social web[C]//Proceedings of the 18th ACM international conference on Multimedia.Irenze,Italy,2010:715-718.
[21] BORTH D,JI R,CHEN T,et al.Large-scale visual sentimentontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM international conference on Multimedia.Barcelona,Spain,2013:223-232.
[22] XU C,CETINTAS S,LEE K C,et al.Visual sentiment prediction with deep convolutional neural networks[J].arXiv:1411.5731.
[23] YOU Q,JIN H,LUO J.Visual sentiment analysis by attending on local image regions[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.San Francisco,California,USA,AAAI Press,2017:231-237.
[24] YANG Y,JIA J,ZHANG S,et al.How do your friends on social media disclose your emotions?[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence.Québec City,Québec,Canada,AAAI Press,2014:306-312.
[25] YUAN J,MCDONOUGH S,YOU Q,et al.Sentribute:image sentiment analysis from a mid-level perspective[C]//Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining(WISDOM 2013).Chicago,IL,USA,ACM,2013:10:1-10:8.
[26] WANG M,CAO D,LI L,et al.Microblog sentiment analysis based on cross-media bag-of-words model[C]//Proceedings of the International Conference on Internet Multimedia Computing and Service.Xiamen,China,ACM,2014:76.
[27] CAO D,JI R,LIN D,et al.A cross-media public sentimentanalysis system for microblog[J].Multimedia Systems,2016,22(4):479-486.
[28] YOU Q,LUO J,JIN H,et al.Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia[C]//Proceedings of the Ninth ACM International Conference on Web Search and Data Mining.San Francisco,CA,USA,ACM,2016:13-22.
[29] YOU Q,CAO L,JIN H,et al.Robust visual-textual sentiment analysis:When attention meets tree-structured recursive neural networks[C]//Proceedings of the 24th ACM International Conference on Multimedia.Amsterdam,Netherlands,ACM,2016:1008-1017.
[30] ZADEH A,CHEN M,PORIA S,et al.Tensor fusion network for multimodal sentiment analysis [C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing(EMNLP 2017).Copenhagen,Denmark,2017:1103-1114.
[31] BAHDANAU D,CHO K,BENGIO Y.Neural machine transla-tion by jointly learning to align and translate[J].arXiv:1409.0473,2014.
[32] VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Advances in Neural Information Processing Systems.2017:5998-6008.
[33] YANG Z,YANG D,DYER C,et al.Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.San Diego,California,USA,2016:1480-1489.
[34] LU J,XIONG C,PARIKH D,et al.Knowing when to look:Adaptive attention via a visual sentinel for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2017).Honolulu,HI,USA,2017:375-383.
[35] 杨琬琪,高阳,周新民,等.多模态张量数据挖掘算法及应用[J].计算机科学,2012,39(1):9-13.
[1] ZHAO Jia-qi, WANG Han-zheng, ZHOU Yong, ZHANG Di, ZHOU Zi-yuan. Remote Sensing Image Description Generation Method Based on Attention and Multi-scale Feature Enhancement [J]. Computer Science, 2021, 48(1): 190-196.
[2] LIU Yang, JIN Zhong. Fine-grained Image Recognition Method Combining with Non-local and Multi-region Attention Mechanism [J]. Computer Science, 2021, 48(1): 197-203.
[3] WANG Rui-ping, JIA Zhen, LIU Chang, CHEN Ze-wei, LI Tian-rui. Deep Interest Factorization Machine Network Based on DeepFM [J]. Computer Science, 2021, 48(1): 226-232.
[4] WANG Run-zheng, GAO Jian, HUANG Shu-hua, TONG Xin. Malicious Code Family Detection Method Based on Knowledge Distillation [J]. Computer Science, 2021, 48(1): 280-286.
[5] PAN Zu-jiang, LIU Ning, ZHANG Wei, WANG Jian-yong. MTHAM:Multitask Disease Progression Modeling Based on Hierarchical Attention Mechanism [J]. Computer Science, 2020, 47(9): 185-189.
[6] ZHAO Wei, LIN Yu-ming, WANG Chao-qiang, CAI Guo-yong. Opinion Word-pairs Collaborative Extraction Based on Dependency Relation Analysis [J]. Computer Science, 2020, 47(8): 164-170.
[7] YUAN Ye, HE Xiao-ge, ZHU Ding-kun, WANG Fu-lee, XIE Hao-ran, WANG Jun, WEI Ming-qiang, GUO Yan-wen. Survey of Visual Image Saliency Detection [J]. Computer Science, 2020, 47(7): 84-91.
[8] LIU Yan, WEN Jing. Complex Scene Text Detection Based on Attention Mechanism [J]. Computer Science, 2020, 47(7): 135-140.
[9] ZHANG Zhi-yang, ZHANG Feng-li, TAN Qi, WANG Rui-jin. Review of Information Cascade Prediction Methods Based on Deep Learning [J]. Computer Science, 2020, 47(7): 141-153.
[10] YU Yi-lin, TIAN Hong-tao, GAO Jian-wei and WAN Huai-yu. Relation Extraction Method Combining Encyclopedia Knowledge and Sentence Semantic Features [J]. Computer Science, 2020, 47(6A): 40-44.
[11] NI Hai-qing, LIU Dan, SHI Meng-yu. Chinese Short Text Summarization Generation Model Based on Semantic-aware [J]. Computer Science, 2020, 47(6): 74-78.
[12] HUANG Yong-tao, YAN Hua. Scene Graph Generation Model Combining Attention Mechanism and Feature Fusion [J]. Computer Science, 2020, 47(6): 133-137.
[13] ZHANG Yan, LI Tian-rui. Review of Comment-oriented Aspect-based Sentiment Analysis [J]. Computer Science, 2020, 47(6): 194-200.
[14] ZHANG Zhi-yang, ZHANG Feng-li, CHEN Xue-qin, WANG Rui-jin. Information Cascade Prediction Model Based on Hierarchical Attention [J]. Computer Science, 2020, 47(6): 201-209.
[15] DENG Yi-jiao, ZHANG Feng-li, CHEN Xue-qin, AI Qing, YU Su-zhe. Collaborative Attention Network Model for Cross-modal Retrieval [J]. Computer Science, 2020, 47(4): 54-59.
Full text



[1] WANG Zhen-chao, HOU Huan-huan and LIAN Rui. Path Optimization Scheme for Restraining Degree of Disorder in CMT[J]. Computer Science, 2018, 45(4): 122 -125 .
[2] XIA Qing-xun and ZHUANG Yi. Remote Attestation Mechanism Based on Locality Principle[J]. Computer Science, 2018, 45(4): 148 -151 .
[3] LUO Xiao-yang, HUO Hong-tao, WANG Meng-si and CHEN Ya-fei. Passive Image-splicing Detection Based on Multi-residual Markov Model[J]. Computer Science, 2018, 45(4): 173 -177 .
[4] ZHAN Yun-jiao, WEI Ou and HU Jun. Formal Description of Requirement of Slats and Flaps Control System for DO-178C Case[J]. Computer Science, 2018, 45(4): 196 -202 .
[5] HOU Yan-e, KONG Yun-feng and DANG Lan-xue. Greedy Randomized Adaptive Search Procedure Algorithm Combining Set Partitioning for Heterogeneous School Bus Routing Problems[J]. Computer Science, 2018, 45(4): 240 -246 .
[6] CAI Li, LIANG Yu, ZHU Yang-yong and HE Jing. History and Development Tendency of Data Quality[J]. Computer Science, 2018, 45(4): 1 -10 .
[7] SUO Yan-feng, WANG Shao-jie, QIN Yu, LI Qiu-xiang, FENG Da-jun and LI Jing-chun. Summary of Security Technology and Application in Industrial Control System[J]. Computer Science, 2018, 45(4): 25 -33 .
[8] CUI Jian-jing, LONG Jun, MIN Er-xue, YU Yang and YIN Jian-ping. Survey on Application of Homomorphic Encryption in Encrypted Machine Learning[J]. Computer Science, 2018, 45(4): 46 -52 .
[9] WANG Zheng-li, XIE Tian, HE Kun and JIN Yan. 0-1 Knapsack Variant with Time Scheduling[J]. Computer Science, 2018, 45(4): 53 -59 .
[10] LI Xiao-xin, LI Jing-jing, HE Lin and LIU Zhi-yong. Robust Face Recognition via Noise Spatial Structure Embedding and High Dimensional Gradient Orientation Embedding[J]. Computer Science, 2018, 45(4): 285 -290 .