计算机科学

计算机科学 ›› 2025, Vol. 52 ›› Issue (11): 157-165.doi: 10.11896/jsjkx.241000016

• 计算机图形学&多媒体 • 上一篇    下一篇

基于颜色增强的多层次特征融合图像情感识别

李晓宇1, 钱艺1, 文益民1,2, 缪裕青1   

  1. 1 桂林电子科技大学广西图像图形与智能处理重点实验室 广西 桂林 541004
    2 桂林旅游学院广西文化和旅游智慧技术重点实验室 广西 桂林 541006
  • 收稿日期:2024-10-08 修回日期:2024-12-22 出版日期:2025-11-15 发布日期:2025-11-06
  • 通讯作者: 钱艺(qyizos@163.com)
  • 作者简介:(22032201021@mails.guet.edu.cn)
  • 基金资助:
    国家自然科学基金(62366011);广西重点研发计划(桂科AB21220023);广西图像图形与智能处理重点实验室项目(GIIP2306);广西自然科学基金重点项目(2024GXNSFDA010066);桂林电子科技大学研究生教育创新计划项目(2023YCXB11)

Multi-level Feature Fusion Image Emotion Recognition Based on Color Enhancement

LI Xiaoyu1, QIAN Yi1, WEN Yimin1,2, MIU Yuqing1   

  1. 1 Guangxi Key Laboratory of Image and Graphic Intelligent Processing,Guilin University of Electronic Technology,Guilin,Guangxi 541004,China
    2 Guangxi Key Laboratory of Culture and Tourism Smart Technology,Guilin Tourism University,Guilin,Guangxi 541006,China
  • Received:2024-10-08 Revised:2024-12-22 Online:2025-11-15 Published:2025-11-06
  • About author:LI Xiaoyu,born in 1999,postgraduate.Her main research interests include computer vision and image emotion re-cognition.
    QIAN Yi,born in 1993,Ph.D.His main research interests include computer vision and medical image analysis.
  • Supported by:
    National Natural Science Foundation of China(62366011),Guangxi Key Research and Development Plan Project(Guike AB21220023),Key Laboratory Project of Guangxi Image Graphics and Intelligent Processing(GIIP2306),Key Project of Guangxi Natural Science Foundation(2024GXNSFDA010066) and Innovation Project of GUET Graduate Education(2023YCXB11).

PDF (PC)

25

摘要： 图像情感识别旨在分析和理解图像内容所传达的情感,其主要挑战在于弥合潜在视觉特征与抽象情感之间的鸿沟。现有深度学习方法大多通过提取不同层次特征来弥合鸿沟,却忽略了颜色特征的重要性。为此,提出了一种基于颜色增强的多层次特征融合图像情感识别方法。通过所设计的颜色增强模块以及多层次特征提取模块,来提取更具代表性的特征表示。其中,颜色增强模块采用颜色矩方式从RGB和HSV颜色空间提取颜色特征,并扩展其维度以丰富情感信息。多层次特征提取模块则引入注意力机制关注图像中的关键区域,并且使用决策融合的方式来减轻高层与低层特征拼接所带来的信息冗余问题。通过在4个公开数据集上进行实验,验证了所提方法能够有效识别图像情感,相较于主流的图像情感识别方法,其性能得到了较大提升。

关键词: 图像情感识别, 深度学习, 多层次特征, 颜色空间, 决策融合

Abstract: Image emotion recognition aims to analyze and understand the emotion conveyed by the content of images.The primary challenge lies in bridging the gap between latent visual features and abstract emotion.Existing deep learning methods mostly bridge this gap by extracting different levels of features,but they often overlook the importance of color features.To address the problem,this paper proposes a multi-level feature fusion image emotion recognition method based on color enhancement.By introducing a color enhancement module and a multi-level feature extraction module,more representative feature representations are extracted.The color enhancement module extracts color features from both the RGB and HSV color spaces using the color moment method,and expands their dimensions to enrich emotional information.The multi-level feature extraction module introduces an attention mechanism to focus on key regions in the image and employs decision fusion to mitigate the issue of information redundancy caused by concatenating high-level and low-level features.Experiments conducted on four public datasets demonstrate that the proposed method can effectively recognize image emotion and significantly improve performance compared to mainstream image emotion recognition methods.

Key words: Image emotion recognition, Deep learning, Multi-level features, Color space, Decision fusion

中图分类号: 

  • TP391
[1]YOU Q,LUO J,JIN H,et al.Building a large scale dataset for image emotion recognition:the fine print and the benchmark [C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.2016:308-314.
[2]MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory [C]//Proceedings of the 18th ACM International Conference on Multimedia.2010:83-92.
[3]JOSHI D,DATTA R,FEDOROVSKAYA E,et al.Aesthetics and emotions in images [J].IEEE Signal Processing Magazine,2011,28(5):94-115.
[4]WANG W,YU Y,ZHANG J.Image emotional classification:static vs.dynamic [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.IEEE,2004:6407-6411.
[5]WANG W,YU Y,JIANG S.Image retrieval by emotional se-mantics:A study of emotional space and feature extraction [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.IEEE,2006:3534-3539.
[6]HE X,ZHAGN W.Emotion recognition by assisted learningwith convolutional neural networks [J].Neurocomputing,2018,291:187-194.
[7]ZHAGN W,YU X,HE X.Learning bidirectional temporal cues for video-based person re-identification [J].IEEE Transactions on Circuits and Systems for Video Technology,2017,28(10):2768-2776.
[8]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.2012.
[9]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [J].arXiv:1409.1556,2014.
[10]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[11]ZHANG W,MA B,LIU K,et al.Video-based pedestrian re-identification by adaptive spatio-temporal appearance model [J].IEEE Transactions on Image Processing,2017,26(4):2042-2054.
[12]LU Y,YUAN C,ZHU W,et al.Structurally incoherent low-rank nonnegative matrix factorization for image classification [J].IEEE Transactions on Image Processing,2018,27(11):5248-5260.
[13]ZHANG W,CHEN Q,ZHANG W,et al.Long-range terrain perception using convolutional neural networks [J].Neurocomputing,2018,275:781-787.
[14]ALEXEY D.An image is worth 16x16 words:Transformers for image recognition at scale [J].arXiv:2010.11929,2020.
[15]BORTHD,CHEN T,JI R,et al.Sentibank:large-scale ontology and classifiers for detecting sentiment and emotions in visual content [C]//Proceedings of the 21st ACM International Conference on Multimedia.2013:459-460.
[16]RAO T,LI X,XU M.Learning multi-level deep representations for image emotion classification [J].Neural Processing Letters,2020,51:2043-2061.
[17]PENG K,CHEN T,SADOVNIK A,et al.A mixed bag of emotions:Model,predict,and transfer emotion distributions [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:860-868.
[18]YANG J,SHE D,SUN M,et al.Visual sentiment predictionbased on automatic discovery of affective regions [J].IEEE Transactions on Multimedia,2018,20(9):2513-2525.
[19]ALAMEDA-PINEDA X,RICCI E,YAN Y,et al.Recognizingemotions from abstract paintings using non-linear matrix completion [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:5240-5248.
[20]ZHANG W,HE X,LU W.Exploring discriminative representations for image emotion recognition with CNNs [J].IEEE Transactions on Multimedia,2019,22(2):515-523.
[21]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2414-2423.
[22]GUILFORD J P,SMITH P C.A system of color-preferences[J].The American Journal of Psychology,1959,72(4):487-502.
[23]JACOBS K W,HUSTMYER JR F E.Effects of four psychological primary colors on GSR,heart rate and respiration rate [J].Perceptual and Motor Skills,1974,38(3):763-766.
[24]STRICKER M A,ORENGO M.Similarity of color images[C]//Proceedings of the Storage and Retrieval for Image and Video Databases III.SPiE,1995:381-392.
[25]DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[26]SHE D,YANG J,CHENG M,et al.WSCNet:Weakly super-vised coupled networks for visual sentiment classification and detection [J].IEEE Transactions on Multimedia,2019,22(5):1358-1371.
[27]ZHANG J,CHEN M,SUN H,et al.Object semantics sentiment correlation analysis enhanced image sentiment classification [J].Knowledge-Based Systems,2020,191:105245.
[28]ZHU X,LI L,ZHANG W,et al.Dependency Exploitation:AUnified CNN-RNN Approach for Visual Emotion Recognition[C]//Proceedings of the International Joint Conference on Artificial Intelligence(IJCAI).2017:3595-3601.
[29]ZHANG H,XU D,LUO G,et al.Learning multi-le- vel representations for affective image recognition [J].Neural Computing and Applications,2022,34(16):14107-14120.
[30]MA J,ZHANG H,HE K,et al.Exploring affective image repre-sentation with visual attention and aesthetic fusion [C]//Proceedings of the Fourteenth International Conference on Graphics and Image Processing(ICGIP 2022).SPIE,2023:603-612.
[31]PATTERSON G,HAYS J.Sun attribute database:Discovering,annotating,and recognizing scene attributes [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2012:2751-2758.
[32]PROFUSEK P J,RAINEY D W.Effects of baker-miller pink and red on state anxiety,grip strength,and motor precision [J].Perceptual and Motor Skills,1987,65(3):941-942.
[33]JACOBS K W,SUESS J F.Effects of four psycho- logical prima-ry colors on anxiety state [J].Perceptual and Motor Skills,1975,41(1):207-210.
[34]MISRA D,NALAMADA T,ARASANIPALAI A U,et al.Rotate to attend:Convolutional triplet attention module [C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021:3139-3148.
[35]SIERSDORFER S,MINACK E,DENG F,et al.Analyzing and predicting sentiment of images on the social web [C]//Proceedings of the 18th ACM International Conference on Multimedia.2010:715-718.
[36]ZHAO S,GAO Y,JIANG X,et al.Exploring principles-of-art features for image emotion recognition [C]//Proceedings of the 22nd ACM International Conference on Multimedia.2014:47-56.
[37]YOU Q,LUO J,JIN H,et al.Robust image sentiment analysis using progressively trained and domain transferred deep networks [C]//Proceedings of the AAAI Conference on Artificial Intelligence.2015.
[38]CHEN T,BORTH D,DARRELL T,et al.Deepsentibank:Visual sentiment concept classification with deep convolutional neural networks [J].arXiv:1410.8586,2014.
[39]YANG J,SHE D,SUN M.Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network [C]//Proceedings of the International Joint Conference on Artificial Intelligence(IJCAI).2017:3266-3272.
[40]ZHANG H,LI H P,PENG G Q,et al.Multi-level feature fusion representation for image emotion recognition [J].Journal of Computer Aided Design and Graphics,2023,35(10):1566-1576.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发