计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (12): 203-211.doi: 10.11896/jsjkx.221100177

• 计算机图形学&多媒体 • 上一篇    下一篇

图像情感分析的层次图卷积网络模型

谈钱辉, 温佳璇, 唐继辉, 孙玉宝   

  1. 南京信息工程大学数字取证教育部工程研究中心 南京 210044
    江苏省大数据分析技术重点实验室 南京 210044
  • 收稿日期:2022-11-21 修回日期:2023-04-28 出版日期:2023-12-15 发布日期:2023-12-07
  • 通讯作者: 孙玉宝(sunyb@nuist.edu.cn)
  • 作者简介:(20201222018@nuist.edu.cn)
  • 基金资助:
    国家重点研发计划(2022YFC2405600);国家自然科学基金(62276139,U2001211)

Hierarchical Graph Convolutional Network for Image Sentiment Analysis

TAN Qianhui, WEN Jiaxuan, TANG Jihui, SUN Yubao   

  1. Digital Forensics Engineering Research Center of the Ministry of Education,Nanjing University of Information Science and Technology,Nanjing 210044,China
    Jiangsu Big Data Analysis Technology Laboratory,Nanjing 210044,China
  • Received:2022-11-21 Revised:2023-04-28 Online:2023-12-15 Published:2023-12-07
  • About author:TAN Qianhui,born in 1998,postgra-duate.His main research interests include deep learning and image sentiment analysis.
    SUN Yubao,born in 1983,Ph.D,professor,isa member of China Computer Federation.His main research interestsinclude deep learning theory and applications and image sentiment analysis.
  • Supported by:
    National Key Research and Development Program of China(2022YFC2405600) and National Natural Science Foundation of China(62276139,U2001211).

PDF (PC)

2163

摘要： 图像情感分析任务旨在运用机器学习模型自动预测观测者对图像的情感反应。当前基于深度网络的情感分析方法广受关注,主要通过卷积神经网络自动学习图像的深度特征。然而,图像情感是图像全局上下文特征的综合反映,由于卷积核感受野的尺寸限制,无法有效捕捉远距离情感特征间的依赖关系,同时网络中不同层次的情感特征间未能得到有效的融合利用,影响了图像情感分析的准确性。为解决上述问题,文中提出了层次图卷积网络模型,分别在空间和通道维度上构建空间上下文图卷积(SCGCN)模块和动态融合图卷积(DFGCN)模块,有效学习不同层次情感特征内部的全局上下文关联与不同层级特征间的关系依赖,能够有效提升情感分类的准确度。网络结构由4个层级预测分支和1个融合预测分支组成,层级预测分支利用SCGCN学习单层次特征的情感上下文表达,融合预测分支利用DFGCN自适应聚合不同语义层次的上下文情感特征,实现融合推理与分类。在4个情感数据集上进行实验,结果表明,所提方法在情感极性分类和细粒度情感分类上的效果均优于现有的图像情感分类模型。

关键词: 图像情感分析, 图卷积, 全局上下文关联, 层次特征关联, 融合分类

Abstract: The image sentiment analysis task aims to use machine learning models to automatically predict the observer's emotional response to images.At present,the sentiment analysis method based on the deep network has attracted wide attention,mainly through the automatic learning of the deep features of the image through the convolutional neural network.However,image emotion is a comprehensive reflection of the global contextual features of the image.Due to the limitation of the receptive field size of the convolution kernel,it is impossible to effectively capture the dependencies between long-distance emotional features.At the same time,the emotional features of different levels in the network cannot be effectively fused and utilized.It affects the accuracy of image sentiment analysis.In order to solve the above problems,this paper proposes a hierarchical graph convolutional network model,and constructs spatial context graph convolution(SCGCN) and dynamic fusion graph convolution(DFGCN).The spatial and channel dimensions are mapped respectively to learn the global context association within different levels of emotional features and the relationship dependence between different levels of features,which could improve the sentiment classification accuracy.The network is composed of four hierarchical prediction branches and one fusion prediction branch.The hierarchical prediction branch uses SCGCN to learn the emotion context expression of single-level features,and the fusion prediction branch uses DFGCN to self-adaptively aggregate the context emotion features of different semantic levels to realize fusion reasoning and classification.Experiment results on four emotion datasets show that the proposed method outperforms existing image emotion classification models in both emotion polarity classification and fine-grained emotion classification.

Key words: Image sentiment analysis, Graph convolution, Global context association, Hierarchical feature association, Fusion classification

中图分类号: 

  • TP37
[1]YOU Q,LUO J,JIN H,et al.Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia[C]//Proceedings of the Ninth ACM International Confe-rence on Web Search and Data Mining.2016:13-22.
[2]ZHAO S,YAO X,YANG J,et al.Affective image content ana-lysis:Two decades review and new perspectives[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(10):6729-6751.
[3]ZHAO S,YAO H,GAO Y,et al.Predicting personalized emotion perceptions of social images[C]//Proceedings of the 24th ACM International Conference on Multimedia.2016:1385-1394.
[4]LANG P J,BRADLEY M M,CUTHBERT B N.International affective picture system(IAPS):affective ratings of pictures and instruction manual.(Rep.No.A-8)[R].2008.
[5]ZHAO S,GAO Y,JIANG X H,et al.Exploring Principles-of-Art Features For Image Emotion Recognition[J].ACM,2014:47-56.
[6]BORTH D,CHEN T,JI R,et al.Sentibank:large-scale ontology and classifiers for detecting sentiment and emotions in visual content[C]//Proceedings of the 21st ACM International Conference on Multimedia.2013:459-460.
[7]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[8]CHEN T,BORTH D,DARRELL T,et al.Deepsentibank:Vi-sual sentiment concept classification with deep convolutional neural networks[J].arXiv:1410.8586,2014.
[9]ZHU X,LI L,ZHANG W,et al.Dependency Exploitation:AUnified CNN-RNN Approach for Visual Emotion Recognition[C]//IJCAI.2017:3595-3601.
[10]PANDA R,ZHANG J,LI H,et al.Contemplating visual emotions:Understanding and overcoming dataset bias[C]//Procee-dings of the European Conference on Computer Vision(ECCV).2018:579-595.
[11]SHE D,YANG J,CHENG M M,et al.Wscnet:Weakly supervised coupled networks for visual sentiment classification and detection[J].IEEE Transactions on Multimedia,2019,22(5):1358-1371.
[12]MIKELS J A,FREDRICKSON B L,LARKIN G R,et al.Emotional category data on images from the International Affective Picture System[J].Behavior Research Methods,2005,37(4):626-630.
[13]YANG J,SHE D,SUN M.Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network[C]//IJCAI.2017:3266-3272.
[14]RAO T,XU M,LIU H,et al.Multi-scale blocks based imageemotion classification using multiple instance learning[C]//2016 IEEE International Conference on Image Processing(ICIP).IEEE,2016:634-638.
[15]SUN M,YANG J,WANG K,et al.Discovering affective regions in deep convolutional neural networks for visual sentiment prediction[C]//2016 IEEE International Conference on Multimedia and Expo(ICME).IEEE,2016:1-6.
[16]YANG J,SHE D,SUN M,et al.Visual sentiment predictionbased on automatic discovery of affective regions[J].IEEE Transactions on Multimedia,2018,20(9):2513-2525.
[17]YOU Q,JIN H,LUO J.Visual sentiment analysis by attending on local image regions[C]//Thirty-First AAAI Conference on Artificial Intelligence.2017.
[18]ZHANG J,CHEN M,SUN H,et al.Object semantics sentiment correlation analysis enhanced image sentiment classification[J].Knowledge-Based Systems,2020,191:105245.
[19]HU Y,WEN G,CHAPMAN A,et al.Graph-based visual-se-mantic entanglement network for zero-shot image recognition[J].IEEE Transactions on Multimedia,2021,24:2473-2487.
[20]CHANDRA S,USUNIER N,KOKKINOS I.Dense and low-rank gaussian crfs using deep embeddings[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:5103-5112.
[21]BERTASIUS G,TORRESANI L,YU S X,et al.Convolutional random walk networks for semantic image segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:858-866.
[22]YAN J D,JIA C Y.Text Classification Method Based on Dou-ble-Graph Neural Network Information Fusion[J].Computer Science,2020,49(8):230-236.
[23]ZHOU F Q,CHENG W Q.Sequence recommendation based on globally enhanced graph neural network[J].Computer Science,2022,49(9):55-63.
[24]ZHOU H Y,ZHANG D Q.Multi-center data-oriented hypergraph convolutional neural network and its application[J].Computer Science,2022,49(3):129-133.
[25]LI Z M,ZHANG Y P,LIU Y J,et al.Point Cloud Representation Learning Based on Deformable Graph Convolution[J].Computer Science,2022,49(8):273-278.
[26]WANG X,GUPTA A.Videos as space-time region graphs[C]//Proceedings of the European conference on computer vision(ECCV).2018:399-417.
[27]CHEN Y,ROHRBACH M,YAN Z,et al.Graph-based globalreasoning networks[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2019:433-442.
[28]LIN T Y,DOLLÁR P,GIRSHICK R,et al.Feature pyramidnetworks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:2117-2125.
[29]LIU D,PURI R,KAMATH N,et al.Composition-aware image aesthetics assessment[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2020:3569-3578.
[30]YOU Q,LUO J,JIN H,et al.Building a large scale dataset for image emotion recognition:The fine print and the benchmark[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2016.
[31]MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM International Conference on Multi-media.2010:83-92.
[32]PENG K C,SADOVNIK A,GALLAGHER A,et al.Where do emotions come from? predicting the emotion stimuli map[C]//2016 IEEE International Conference on Image Processing(ICIP).IEEE,2016:614-618.
[33]YOU Q,LUO J,JIN H,et al.Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Twenty-ninth AAAI Conference on Artificial Intelligence.2015.
[34]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[35]JIE H,LI S,GANG S.Squeeze-and-Excitation Networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2018.
[36]WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional Block Attention Module[C]//Proceedings of the European Confe-rence on Computer Vision(ECCV).2018:3-19.
[37]WANG Q,WU B,ZHU P,et al.ECA-Net:Efficient Channel Attention for Deep Convolutional Neural Networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2020.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发