Computer Science ›› 2020, Vol. 47 ›› Issue (11A): 508-514.doi: 10.11896/jsjkx.191100041

• Big Data & Data Science • Previous Articles     Next Articles

Multimodal Sentiment Analysis Based on Attention Neural Network

LIN Min-hong, MENG Zu-qiang   

  1. School of Computer,Electronics and Information,Guangxi University,Nanning 530004,China
  • Online:2020-11-15 Published:2020-11-17
  • About author:LIN Min-hong,born in 1995,postgraduate.Her main research interests include data mining and cross-media mining.
    MENG Zu-qiang,born in 1974,Ph.D,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include data mining,cross-media mining and KDD (Knowledge Discovery in Database).
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61762009).

Abstract: In recent years,more and more people are keen to express their feelings and opinions in the form of both pictures and texts on social media,and the scale of multimodal data including images and texts keeps growing.Compared with single mode data,multimodal data contains more information.It can better reveal the real emotion of users.Sentiment analysis of these huge amounts of multimodal data helps to better understand people's attitudes and opinions.In addition,it has a wide range of applications.In order to solve the problem of information redundancy in multimodal sentiment analysis task,this paper proposes a multimodal sentiment analysis method based on tensor fusion scheme and attention neural network.This method constructs the text feature extraction model and image feature extraction model based on attention neural network to highlight the key areas of image emotion information and words containing emotion information,so as to make the expression of each feature more concise and accurate.It fuses each modal feature using tensor fusion method in order to obtain the joint feature vector.Finally,it uses support vector machine for sentiment classification.The experimental results of this model on two real Twitter data sets show that compared with other sentiment analysis models,this method has a great improvement in precision rate,recall rate,F1 score andaccuracy rate.

Key words: Attention mechanism, Multimodal data, Sentiment analysis, Social media, Tensor fusion

CLC Number: 

  • TP391
[1] ASUR S,HUBERMAN B A.Predicting the future with socialmedia [C] //Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.Toronto,Canada,2010:492-499.
[2] O'CONNOR B,BALASUBRAMANYAN R,ROUTLEDGE B R,et al.From tweets to polls:Linking text sentiment to public opinion time series [C]//Proceedings of the International AAAI Conference on Weblogs And Social Media.Washington,United States,2010:11:122-129.
[3] TUMASJAN A,SPRENGER T O,SANDNER P G,et al.Predicting elections with twitter:What 140 characters reveal about political sentiment[C]//Proceedings of the International AAAI Conference on Weblogs And Social Media.Washington,USA,2010:10:122-129.
[4] WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional Block Attention Module[C]//Proceedings of the European Conference on Computer Vision 2018.ECCV,Munich,Germany,2018:3-19.
[5] LI X,XIE H,CHEN L,et al.News impact on stock price return via sentiment analysis[J].Knowledge-Based Systems,2014,69:14-23.
[6] NGUYEN T H,SHIRAI K.Topic modeling based sentimentanalysis on social media for stock market prediction[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).Beijing,China,2015:1354-1364.
[7] ZHANG L,ZHAO Y,ZHU Z F.Advances in SemanticallyShared Subspace Learning for Cross-Media Data[J].Chinese Journal of Computers,2017.
[8] ZHANG L,WANG S,LIU B.Deep learning for sentiment analysis:A survey[J].Wiley Interdisciplinary Reviews:Data Mining and Knowledge Discovery,2018:e1253.
[9] TURNEY P D.Thumbs up or thumbs down:semantic orientation applied to unsupervised classification of reviews[J].Proceedings of Annual Meeting of the Association for Computational Linguistics,2002:417-424.
[10] TABOADAM,BROOKE J,TOFILOSKI M,et al.Lexicon-Based Methods for Sentiment Analysis[J].Computational Linguistics,2011,37(2):267-307.
[11] BACCIANELLA S,ESULI A,SEBASTIANI F.Sentiwordnet3.0:an enhanced lexical resource for sentiment analysis and opinion mining[C]//Proceedings of the International Conference on Language Resources and Evaluation.Valletta,Malta,European.2010,2010(10):2200-2204.
[12] HU M,LIU B.Mining and summarizing customer reviews[C]//Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.ACM,2004:168-177.
[13] PANG B,LEE L,VAITHYANATHAN S.Thumbs up?:sentiment classification using machine learning techniques [C]//Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing.2002.
[14] KIM Y.Convolutional neural networks for sentence classification[J].arXiv,2014:1408.5882.
[15] KALCHBRENNER N,GREFENSTETTE E,BLUNSOM P.A Convolutional Neural Network for Modelling Sentences[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Association for Computational Linguistics,2014:655-665.
[16] TAY Y,TUAN L A,HUI S C.Learning to attend via word-as-pect associative fusion for aspect-based sentiment analysis[C]//Thirty-Second AAAI Conference on Artificial Intelligence,Louisiana,USA,2018.
[17] TANG D,QIN B,LIU T.Document modeling with gated re-current neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.Lisbon,Portugal,2015:1422-1432.
[18] SONG J,YU Q,SONG Y Z,et al.Deep spatial-semantic attention for fine-grained sketch-based image retrieval[C]//Proceedings of the IEEE International Conference on Computer Vision(ICCV 2017).Venice,Italy,2017:5551-5560.
[19] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[20] SIERSDORFER S,MINACK E,DENG F,et al.Analyzing and predicting sentiment of images on the social web[C]//Proceedings of the 18th ACM international conference on Multimedia.Irenze,Italy,2010:715-718.
[21] BORTH D,JI R,CHEN T,et al.Large-scale visual sentimentontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM international conference on Multimedia.Barcelona,Spain,2013:223-232.
[22] XU C,CETINTAS S,LEE K C,et al.Visual sentiment prediction with deep convolutional neural networks[J].arXiv:1411.5731.
[23] YOU Q,JIN H,LUO J.Visual sentiment analysis by attending on local image regions[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence.San Francisco,California,USA,AAAI Press,2017:231-237.
[24] YANG Y,JIA J,ZHANG S,et al.How do your friends on social media disclose your emotions?[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence.Québec City,Québec,Canada,AAAI Press,2014:306-312.
[25] YUAN J,MCDONOUGH S,YOU Q,et al.Sentribute:image sentiment analysis from a mid-level perspective[C]//Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining(WISDOM 2013).Chicago,IL,USA,ACM,2013:10:1-10:8.
[26] WANG M,CAO D,LI L,et al.Microblog sentiment analysis based on cross-media bag-of-words model[C]//Proceedings of the International Conference on Internet Multimedia Computing and Service.Xiamen,China,ACM,2014:76.
[27] CAO D,JI R,LIN D,et al.A cross-media public sentimentanalysis system for microblog[J].Multimedia Systems,2016,22(4):479-486.
[28] YOU Q,LUO J,JIN H,et al.Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia[C]//Proceedings of the Ninth ACM International Conference on Web Search and Data Mining.San Francisco,CA,USA,ACM,2016:13-22.
[29] YOU Q,CAO L,JIN H,et al.Robust visual-textual sentiment analysis:When attention meets tree-structured recursive neural networks[C]//Proceedings of the 24th ACM International Conference on Multimedia.Amsterdam,Netherlands,ACM,2016:1008-1017.
[30] ZADEH A,CHEN M,PORIA S,et al.Tensor fusion network for multimodal sentiment analysis [C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing(EMNLP 2017).Copenhagen,Denmark,2017:1103-1114.
[31] BAHDANAU D,CHO K,BENGIO Y.Neural machine transla-tion by jointly learning to align and translate[J].arXiv:1409.0473,2014.
[32] VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Advances in Neural Information Processing Systems.2017:5998-6008.
[33] YANG Z,YANG D,DYER C,et al.Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.San Diego,California,USA,2016:1480-1489.
[34] LU J,XIONG C,PARIKH D,et al.Knowing when to look:Adaptive attention via a visual sentinel for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2017).Honolulu,HI,USA,2017:375-383.
[35] 杨琬琪,高阳,周新民,等.多模态张量数据挖掘算法及应用[J].计算机科学,2012,39(1):9-13.
[1] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[3] ZHOU Xu, QIAN Sheng-sheng, LI Zhang-ming, FANG Quan, XU Chang-sheng. Dual Variational Multi-modal Attention Network for Incomplete Social Event Classification [J]. Computer Science, 2022, 49(9): 132-138.
[4] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[5] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[6] XIONG Li-qin, CAO Lei, LAI Jun, CHEN Xi-liang. Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization [J]. Computer Science, 2022, 49(9): 172-182.
[7] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[8] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[9] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[10] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[11] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[12] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[13] ZHANG Ying-tao, ZHANG Jie, ZHANG Rui, ZHANG Wen-qiang. Photorealistic Style Transfer Guided by Global Information [J]. Computer Science, 2022, 49(7): 100-105.
[14] ZENG Zhi-xian, CAO Jian-jun, WENG Nian-feng, JIANG Guo-quan, XU Bin. Fine-grained Semantic Association Video-Text Cross-modal Entity Resolution Based on Attention Mechanism [J]. Computer Science, 2022, 49(7): 106-112.
[15] XU Ming-ke, ZHANG Fan. Head Fusion:A Method to Improve Accuracy and Robustness of Speech Emotion Recognition [J]. Computer Science, 2022, 49(7): 132-141.
Full text



No Suggested Reading articles found!