Computer Science ›› 2022, Vol. 49 ›› Issue (1): 219-224.doi: 10.11896/jsjkx.201000074

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Image-Text Sentiment Analysis Model Based on Visual Aspect Attention

YUAN Jing-ling, DING Yuan-yuan, SHENG De-ming, LI Lin   

  1. School of Computer Science and Technology,Wuhan University of Technology,Wuhan 430070,China
  • Received:2020-10-14 Revised:2021-04-19 Online:2022-01-15 Published:2022-01-18
  • About author:YUAN Jing-ling,born in 1975,doctor,is a member of China Computer Federation.Her main research interests include mechine learning,intelligent ana-lysis and green computing.
  • Supported by:
    National Key Research and Development Program of China(2017YFB0802303) and National Natural Science Foundation of China(62076127,61571226).

Abstract: Social network has become an integral part of our daily life.Sentiment analysis of social media information is helpful to understand people's views,attitudes and emotions on social networking sites.Traditional sentiment analysis mainly relies on text.With the rise of smart phones,information on the network is gradually diversified,including not only text,but also images.It is found that,in many cases,images can enhance the text rather than express emotions independently.We propose a novel image text sentiment analysis model (LSTM-VistaNet).Specifically,this model does not take the picture information as the direct input,but uses the VGG16 network to extract the image features,and then generates the visual aspect attention,and gives the core sentences in the document a higher weight,and get a document representation based on the visual aspect attention.In addition,we use the LSTM network to extract the text sentiment and get the document representation based on text only.Finally,we fuse the two groups of classification results to obtain the final classification label.On the Yelp restaurant reviews data set,our model achieves an accuracy of 62.08%,which is 18.92% higher than BiGRU-mVGG,which verifies the effectiveness of using visual information as aspect attention assisted text for emotion classification;It is 0.32% higher than VistaNet model,which proves that LSTM model can effectively make up for the defect that images in VistaNet model cannot completely cover text.

Key words: LSTM, Multimodel, Sentiment analysis, Social images, Visual aspect attention

CLC Number: 

  • TP391.1
[1]LI X,XIE H,CHEN L,et al.News impact on stock price return via sentiment analysis[J].Proceedings of the Knowledeg Based System,2014,69:14-23.
[2]KAGAN V,STEVENS A,SUBRAHMANIAN V S.Usingtwitter sentiment to forecast the 2013 Pakistani election and the 2014 Indian election[J].Proceedings of the IEEE Intelligent Systems,2015,30(1):2-5.
[3]YADAV S,EKBAL A,SAHA S,et al.Medical sentiment analysis using social media:towards building a patient assisted system[C]//Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).2018:2790-2797.
[4]TRUONG T Q,LAUW H W.VistaNet:Visual Aspect Atten-tion Network for Multimodal Sentiment Analysis[C]//Procee-dings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-2019).2019:305-312.
[5]YU D,FU J,MEI T,et al.Multi-level attention networks for visual question answering[C]//Proceedings of the Computer Vision and Pattern Recognition (CVPR).2017:4187-4195.
[6]PORIA S,CHATURVEDI I,CAMBRIA E,et al.Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis[C]//Proceedings of 2016 IEEE 16th International Conference on Data Mining (ICDM).2017:439-448.
[7]XU N,MAO W,CHEN G.Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis[C]//Procee-dings of the AAAI Conference on Artificial Intelligence,2019,33(1),371-378.
[8]CHANSW K,CHONGMW C.Sentiment analysis in financial texts[J].Proceedings of the Decision Support Systems,2017,94(2017):53-64.
[9]SEVERYN A,MOSCHITTI A.Twitter sentiment analysis with deep convolutional neural networks[C]//Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval.2015:959-962.
[10]LAI S,XU L,LIU K,et al.Recurrent convolutional neural networks for text classification[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence.2015:2267-2273.
[11]NGUYEN T,KAVURI S,LEE M.A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips[J].Proceedings of the Neural Networks.2019,118:208-219.
[12]BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv:1409.0473,2014.
[13]YANG Z,YANG D,DYER C,et al.Hierarchical attention networks for document classification[C]//Proceedings of North American Chapter of the Association for Computational Linguistics (HLT-NAACL).2016:1480-1489.
[14]TANG D,QIN B,LIU T.Aspect level sentiment classification with deep memory network[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).2016:214-224.
[15]TRUONG Q,LAUW H W.Visual sentiment analysis for review images with item-oriented and user-oriented CNN[C]//Proceedings of the 25th ACM International Conference on Multimedia.2017:1274-1282.
[16]LI M,GAN T,LIU M,et al.Long-tail Hashtag Recommendation for Micro-videos with Graph Convolutional Network[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.2019:509-518.
[17]SIERSDORFER S,MINACK E,DENG F,et al.Analyzing and predicting sentiment of images on the social web[C]//Procee-dings of the 18th ACM International Conference on Multimedia.2010:715-718.
[18]YOU Q,LUO J,JIN H,et al.Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Proceedings In AAAI.2015:381-388.
[19]BORTH D,JI R,CHEN T,et al.Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Procee-dings of the 21st ACM International Conference on Multimedia.2013:223-232.
[20]YOU Q,JIN H,LUO J.Visual sentiment analysis by attending on local image regions[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence.2017:231-237.
[21]XU N,MAO W.MultiSentiNet:A Deep Semantic Network for Multimodal Sentiment Analysis[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management.2017:2399-2402.
[22]PENNINGTON J,SOCHER R,MANNING C D.Glove:Global vectors for word representation[C]//Proceedings of theConfe-rence on Empirical Methods in Natural Language Processing.2014:1532-1543.
[23]CHO K,VAN M,GULCEHRE C,et al.Learning phrase representations using rnn encoder-decoder for statistical machine translation[C]//Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014).2014:1724-1734.
[24]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the International Conference on Learning Representations.2015.
[25]YUE W,WAEL A,PREMKUMAR N.Multi-Modality ImageManipu-lation Detection [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:9543-9552.
[26]TANG D,QIN B,LIU T.Document modeling with gated recurrent neural network for sentiment classification[C]//Procee-dings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).2015:1422-1432.
[1] ZHANG Yuan, KANG Le, GONG Zhao-hui, ZHANG Zhi-hong. Related Transaction Behavior Detection in Futures Market Based on Bi-LSTM [J]. Computer Science, 2022, 49(7): 31-39.
[2] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[3] LIN Xi, CHEN Zi-zhuo, WANG Zhong-qing. Aspect-level Sentiment Classification Based on Imbalanced Data and Ensemble Learning [J]. Computer Science, 2022, 49(6A): 144-149.
[4] WANG Shan, XU Chu-yi, SHI Chun-xiang, ZHANG Ying. Study on Cloud Classification Method of Satellite Cloud Images Based on CNN-LSTM [J]. Computer Science, 2022, 49(6A): 675-679.
[5] DING Feng, SUN Xiao. Negative-emotion Opinion Target Extraction Based on Attention and BiLSTM-CRF [J]. Computer Science, 2022, 49(2): 223-230.
[6] HU Yan-li, TONG Tan-qian, ZHANG Xiao-yu, PENG Juan. Self-attention-based BGRU and CNN for Sentiment Analysis [J]. Computer Science, 2022, 49(1): 252-258.
[7] DAI Hong-liang, ZHONG Guo-jin, YOU Zhi-ming , DAI Hong-ming. Public Opinion Sentiment Big Data Analysis Ensemble Method Based on Spark [J]. Computer Science, 2021, 48(9): 118-124.
[8] ZHANG Jin, DUAN Li-guo, LI Ai-ping, HAO Xiao-yan. Fine-grained Sentiment Analysis Based on Combination of Attention and Gated Mechanism [J]. Computer Science, 2021, 48(8): 226-233.
[9] CHENG Si-wei, GE Wei-yi, WANG Yu, XU Jian. BGCN:Trigger Detection Based on BERT and Graph Convolution Network [J]. Computer Science, 2021, 48(7): 292-298.
[10] YU Jian-ye, QI Yong, WANG Bao-zhuo. Distributed Combination Deep Learning Intrusion Detection Method for Internet of Vehicles Based on Spark [J]. Computer Science, 2021, 48(6A): 518-523.
[11] HU Yu-wen. Stock Forecast Based on Optimized LSTM Model [J]. Computer Science, 2021, 48(6A): 151-157.
[12] SHI Wei, FU Yue. Microblog Short Text Mining Considering Context:A Method of Sentiment Analysis [J]. Computer Science, 2021, 48(6A): 158-164.
[13] CHEN Hui-qin, GUO Guan-cheng, QIN Chao-xuan, LI Zhao-bi. Research on Elderly Population Prediction Based on GM-LSTM Model in Nanjing City [J]. Computer Science, 2021, 48(6A): 231-234.
[14] PAN Fang, ZHANG Hui-bing, DONG Jun-chao, SHOU Zhao-yu. Aspect Sentiment Analysis of Chinese Online Course Review Based on Efficient Transformer [J]. Computer Science, 2021, 48(6A): 264-269.
[15] ZHANG Zheng-wan, WU Di, ZHANG Chun-jiong. Study of Cellular Traffic Prediction Based on Multi-channel Sparse LSTM [J]. Computer Science, 2021, 48(6): 296-300.
Full text



No Suggested Reading articles found!