Computer Science ›› 2020, Vol. 47 ›› Issue (11): 250-254.doi: 10.11896/jsjkx.190800154

• Artificial Intelligence • Previous Articles     Next Articles

Visual Sentiment Prediction with Visual Semantic Embedding and Attention Mechanism

LAN Yi-lun, MENG Min, WU Ji-gang   

  1. Department of Computer Science,Guangdong University of Technology,Guangzhou 510006,China
  • Received:2019-08-29 Revised:2019-11-22 Online:2020-11-15 Published:2020-11-05
  • About author:LAN Yi-lun,born in 1995,postgra-duate.His main research interests include visual sentiment prediction and image classification
    MENG Min,born in 1985,Ph.D,asso-ciate professor,postgraduate supervisor,is a member of China Computer Federation.Her main research interests include image processing and machine learning.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61702114) and Guangdong Key R&D Project of China (2019B010121001).

Abstract: In order to bridge the semantic gap between visual features and sentiments and reduce the impact of sentiment irrelevant regions in the image,this paper presents a novel visual sentiment prediction method by integrating visual semantic embedding and attention mechanism.Firstly,the method employs the auto-encoder to learn joint embedding of image features and semantic features,so as to alleviate the difference between the low-level visual features and the high-level semantic features.Secondly,a set of salient region features are extracted as input to the attention model,in which the correlations between salient regions and joint embedding features can be established to discover sentiment relevant regions.Finally,the sentiment classifier is built on top of these regions for visual sentiment prediction.The experimental results show that,the proposed method significantly improves the classification performance on testing samples and outperforms the state-of-the-art algorithms on visual sentiment analysis.

Key words: Attention mechanism, Salient regions detection, Visual semantic embedding, Visual sentiment prediction

CLC Number: 

  • TP391.41
[1] PANG B,LEE L.Opinion mining and sentiment analysis[J].Foundations and Trends® in Information Retrieval,2008,2(1/2):1-135.
[2] YANG J,SHE D,SUN M,et al.Visual sentiment predictionbased on automatic discovery of affective regions[J].IEEE Transactions on Multimedia,2018,20(9):2513-2525.
[3] YOU Q,JIN H,LUO J.Visual sentiment analysis by attendingon local image regions[C]//Thirty-First AAAI Conference on Artificial Intelligence.2017:231-237.
[4] SONG K,YAO T,LING Q,et al.Boosting image sentimentanalysis with visual attention[J].Neurocomputing,2018,312:218-228.
[5] FAN S,JIANG M,SHEN Z,et al.The Role of Visual Attention in Sentiment Prediction[C]//Proceedings of the 25th ACM International Conference on Multimedia.ACM,2017:217-225.
[6] FAN S,SHEN Z,JIANG M,et al.Emotional attention:A study of image sentiment and visual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7521-7531.
[7] ANDERSON P,HE X,BUEHLER C,et al.Bottom-up and top-down attention for image captioning and visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:6077-6086.
[8] WEI-NING W,YING-LIN Y,SHENG-MING J.Image retrieval by emotional semantics:A study of emotional space and feature extraction[C]//2006 IEEE International Conference on Systems,Man and Cybernetics.IEEE,2006,4:3534-3539.
[9] MACHAJDIK J,HANBURY A.Affective image classificationusing features inspired by psychology and art theory[C]//Proceedings of the 18th ACM international conference on Multimedia.ACM,2010:83-92.
[10] ZHAO S,GAO Y,JIANG X,et al.Exploring principles-of-art features for image emotion recognition[C]//Proceedings of the 22nd ACM international conference on Multimedia.ACM,2014:47-56.
[11] BORTH D,JI R,CHEN T,et al.Large-scale visual sentimentontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM International Conference on Multimedia.ACM,2013:223-232.
[12] LI Z,FAN Y,LIU W,et al.Image sentiment prediction based on textual descriptions with adjective noun pairs[J].Multimedia Tools and Applications,2018,77(1):1115-1132.
[13] CAMPOS V,JOU B,GIRO-I-NIETO X.From pixels to sentiment:Fine-tuning CNNs for visual sentiment prediction[J].Ima-ge and Vision Computing,2017,65:15-22.
[14] DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition.IEEE,2009:248-255.
[15] ZHU X,LI L,ZHANG W,et al.Dependency Exploitation:AUnified CNN-RNN Approach for Visual Emotion Recognition[C]//IJCAI.2017:3595-3601.
[16] BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy layer-wise training of deep networks[C]//Advances in Neural Information Processing Systems.2007:153-160.
[17] VINCENT P,LAROCHELLE H,LAJOIE I,et al.Stacked denoising autoencoders:Learning useful representations in a deep network with a local denoising criterion[J].Journal of Machine Learning Research,2010,11(Dec):3371-3408.
[18] XU K,BA J,KIROS R,et al.Show,attend and tell:Neural image caption generation with visual attention[C]//International Conference on Machine Learning.2015:2048-2057.
[19] FAN S,NG T T,HERBERG J S,et al.An automated estimator of image visual realism based on human cognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:4201-4208.
[20] CHEN T,BORTH D,DARRELL T,et al.Deepsentibank:Visual sentiment concept classification with deep convolutional neural networks[J].arXiv:1410.8586,2014.
[21] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convol-utional neural networks[C]//Advances in Neural Information Processing Systems.2012:1097-1105.
[22] YOU Q,CAO L,JIN H,et al.Robust visual-textual sentiment analysis:When attention meets tree-structured recursive neural networks[C]//Proceedings of the 24th ACM International Conference on Multimedia.ACM,2016:1008-1017.
[23] CAMPOS V,JOU B,GIRO-I-NIETO X.From pixels to sentiment:Fine-tuning CNNs for visual sentiment prediction[J].Ima-ge and Vision Computing,2017,65:15-22.
[1] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[2] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[3] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[4] XIONG Li-qin, CAO Lei, LAI Jun, CHEN Xi-liang. Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization [J]. Computer Science, 2022, 49(9): 172-182.
[5] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[8] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[9] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[10] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[11] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[12] XIONG Luo-geng, ZHENG Shang, ZOU Hai-tao, YU Hua-long, GAO Shang. Software Self-admitted Technical Debt Identification with Bidirectional Gate Recurrent Unit and Attention Mechanism [J]. Computer Science, 2022, 49(7): 212-219.
[13] PENG Shuang, WU Jiang-jiang, CHEN Hao, DU Chun, LI Jun. Satellite Onboard Observation Task Planning Based on Attention Neural Network [J]. Computer Science, 2022, 49(7): 242-247.
[14] ZHANG Ying-tao, ZHANG Jie, ZHANG Rui, ZHANG Wen-qiang. Photorealistic Style Transfer Guided by Global Information [J]. Computer Science, 2022, 49(7): 100-105.
[15] ZENG Zhi-xian, CAO Jian-jun, WENG Nian-feng, JIANG Guo-quan, XU Bin. Fine-grained Semantic Association Video-Text Cross-modal Entity Resolution Based on Attention Mechanism [J]. Computer Science, 2022, 49(7): 106-112.
Full text



No Suggested Reading articles found!