Computer Science ›› 2025, Vol. 52 ›› Issue (11): 157-165.doi: 10.11896/jsjkx.241000016

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Multi-level Feature Fusion Image Emotion Recognition Based on Color Enhancement

LI Xiaoyu1, QIAN Yi1, WEN Yimin1,2, MIU Yuqing1   

  1. 1 Guangxi Key Laboratory of Image and Graphic Intelligent Processing,Guilin University of Electronic Technology,Guilin,Guangxi 541004,China
    2 Guangxi Key Laboratory of Culture and Tourism Smart Technology,Guilin Tourism University,Guilin,Guangxi 541006,China
  • Received:2024-10-08 Revised:2024-12-22 Online:2025-11-15 Published:2025-11-06
  • About author:LI Xiaoyu,born in 1999,postgraduate.Her main research interests include computer vision and image emotion re-cognition.
    QIAN Yi,born in 1993,Ph.D.His main research interests include computer vision and medical image analysis.
  • Supported by:
    National Natural Science Foundation of China(62366011),Guangxi Key Research and Development Plan Project(Guike AB21220023),Key Laboratory Project of Guangxi Image Graphics and Intelligent Processing(GIIP2306),Key Project of Guangxi Natural Science Foundation(2024GXNSFDA010066) and Innovation Project of GUET Graduate Education(2023YCXB11).

Abstract: Image emotion recognition aims to analyze and understand the emotion conveyed by the content of images.The primary challenge lies in bridging the gap between latent visual features and abstract emotion.Existing deep learning methods mostly bridge this gap by extracting different levels of features,but they often overlook the importance of color features.To address the problem,this paper proposes a multi-level feature fusion image emotion recognition method based on color enhancement.By introducing a color enhancement module and a multi-level feature extraction module,more representative feature representations are extracted.The color enhancement module extracts color features from both the RGB and HSV color spaces using the color moment method,and expands their dimensions to enrich emotional information.The multi-level feature extraction module introduces an attention mechanism to focus on key regions in the image and employs decision fusion to mitigate the issue of information redundancy caused by concatenating high-level and low-level features.Experiments conducted on four public datasets demonstrate that the proposed method can effectively recognize image emotion and significantly improve performance compared to mainstream image emotion recognition methods.

Key words: Image emotion recognition, Deep learning, Multi-level features, Color space, Decision fusion

CLC Number: 

  • TP391
[1]YOU Q,LUO J,JIN H,et al.Building a large scale dataset for image emotion recognition:the fine print and the benchmark [C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.2016:308-314.
[2]MACHAJDIK J,HANBURY A.Affective image classification using features inspired by psychology and art theory [C]//Proceedings of the 18th ACM International Conference on Multimedia.2010:83-92.
[3]JOSHI D,DATTA R,FEDOROVSKAYA E,et al.Aesthetics and emotions in images [J].IEEE Signal Processing Magazine,2011,28(5):94-115.
[4]WANG W,YU Y,ZHANG J.Image emotional classification:static vs.dynamic [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.IEEE,2004:6407-6411.
[5]WANG W,YU Y,JIANG S.Image retrieval by emotional se-mantics:A study of emotional space and feature extraction [C]//Proceedings of the IEEE International Conference on Systems,Man and Cybernetics.IEEE,2006:3534-3539.
[6]HE X,ZHAGN W.Emotion recognition by assisted learningwith convolutional neural networks [J].Neurocomputing,2018,291:187-194.
[7]ZHAGN W,YU X,HE X.Learning bidirectional temporal cues for video-based person re-identification [J].IEEE Transactions on Circuits and Systems for Video Technology,2017,28(10):2768-2776.
[8]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.2012.
[9]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition [J].arXiv:1409.1556,2014.
[10]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[11]ZHANG W,MA B,LIU K,et al.Video-based pedestrian re-identification by adaptive spatio-temporal appearance model [J].IEEE Transactions on Image Processing,2017,26(4):2042-2054.
[12]LU Y,YUAN C,ZHU W,et al.Structurally incoherent low-rank nonnegative matrix factorization for image classification [J].IEEE Transactions on Image Processing,2018,27(11):5248-5260.
[13]ZHANG W,CHEN Q,ZHANG W,et al.Long-range terrain perception using convolutional neural networks [J].Neurocomputing,2018,275:781-787.
[14]ALEXEY D.An image is worth 16x16 words:Transformers for image recognition at scale [J].arXiv:2010.11929,2020.
[15]BORTHD,CHEN T,JI R,et al.Sentibank:large-scale ontology and classifiers for detecting sentiment and emotions in visual content [C]//Proceedings of the 21st ACM International Conference on Multimedia.2013:459-460.
[16]RAO T,LI X,XU M.Learning multi-level deep representations for image emotion classification [J].Neural Processing Letters,2020,51:2043-2061.
[17]PENG K,CHEN T,SADOVNIK A,et al.A mixed bag of emotions:Model,predict,and transfer emotion distributions [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:860-868.
[18]YANG J,SHE D,SUN M,et al.Visual sentiment predictionbased on automatic discovery of affective regions [J].IEEE Transactions on Multimedia,2018,20(9):2513-2525.
[19]ALAMEDA-PINEDA X,RICCI E,YAN Y,et al.Recognizingemotions from abstract paintings using non-linear matrix completion [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:5240-5248.
[20]ZHANG W,HE X,LU W.Exploring discriminative representations for image emotion recognition with CNNs [J].IEEE Transactions on Multimedia,2019,22(2):515-523.
[21]GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2414-2423.
[22]GUILFORD J P,SMITH P C.A system of color-preferences[J].The American Journal of Psychology,1959,72(4):487-502.
[23]JACOBS K W,HUSTMYER JR F E.Effects of four psychological primary colors on GSR,heart rate and respiration rate [J].Perceptual and Motor Skills,1974,38(3):763-766.
[24]STRICKER M A,ORENGO M.Similarity of color images[C]//Proceedings of the Storage and Retrieval for Image and Video Databases III.SPiE,1995:381-392.
[25]DENG J,DONG W,SOCHER R,et al.Imagenet:A large-scale hierarchical image database [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[26]SHE D,YANG J,CHENG M,et al.WSCNet:Weakly super-vised coupled networks for visual sentiment classification and detection [J].IEEE Transactions on Multimedia,2019,22(5):1358-1371.
[27]ZHANG J,CHEN M,SUN H,et al.Object semantics sentiment correlation analysis enhanced image sentiment classification [J].Knowledge-Based Systems,2020,191:105245.
[28]ZHU X,LI L,ZHANG W,et al.Dependency Exploitation:AUnified CNN-RNN Approach for Visual Emotion Recognition[C]//Proceedings of the International Joint Conference on Artificial Intelligence(IJCAI).2017:3595-3601.
[29]ZHANG H,XU D,LUO G,et al.Learning multi-le- vel representations for affective image recognition [J].Neural Computing and Applications,2022,34(16):14107-14120.
[30]MA J,ZHANG H,HE K,et al.Exploring affective image repre-sentation with visual attention and aesthetic fusion [C]//Proceedings of the Fourteenth International Conference on Graphics and Image Processing(ICGIP 2022).SPIE,2023:603-612.
[31]PATTERSON G,HAYS J.Sun attribute database:Discovering,annotating,and recognizing scene attributes [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2012:2751-2758.
[32]PROFUSEK P J,RAINEY D W.Effects of baker-miller pink and red on state anxiety,grip strength,and motor precision [J].Perceptual and Motor Skills,1987,65(3):941-942.
[33]JACOBS K W,SUESS J F.Effects of four psycho- logical prima-ry colors on anxiety state [J].Perceptual and Motor Skills,1975,41(1):207-210.
[34]MISRA D,NALAMADA T,ARASANIPALAI A U,et al.Rotate to attend:Convolutional triplet attention module [C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021:3139-3148.
[35]SIERSDORFER S,MINACK E,DENG F,et al.Analyzing and predicting sentiment of images on the social web [C]//Proceedings of the 18th ACM International Conference on Multimedia.2010:715-718.
[36]ZHAO S,GAO Y,JIANG X,et al.Exploring principles-of-art features for image emotion recognition [C]//Proceedings of the 22nd ACM International Conference on Multimedia.2014:47-56.
[37]YOU Q,LUO J,JIN H,et al.Robust image sentiment analysis using progressively trained and domain transferred deep networks [C]//Proceedings of the AAAI Conference on Artificial Intelligence.2015.
[38]CHEN T,BORTH D,DARRELL T,et al.Deepsentibank:Visual sentiment concept classification with deep convolutional neural networks [J].arXiv:1410.8586,2014.
[39]YANG J,SHE D,SUN M.Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network [C]//Proceedings of the International Joint Conference on Artificial Intelligence(IJCAI).2017:3266-3272.
[40]ZHANG H,LI H P,PENG G Q,et al.Multi-level feature fusion representation for image emotion recognition [J].Journal of Computer Aided Design and Graphics,2023,35(10):1566-1576.
[1] YIN Shi, SHI Zhenyang, WU Menglin, CAI Jinyan, YU De. Deep Learning-based Kidney Segmentation in Ultrasound Imaging:Current Trends and Challenges [J]. Computer Science, 2025, 52(9): 16-24.
[2] ZENG Lili, XIA Jianan, LI Shaowen, JING Maike, ZHAO Huihui, ZHOU Xuezhong. M2T-Net:Cross-task Transfer Learning Tongue Diagnosis Method Based on Multi-source Data [J]. Computer Science, 2025, 52(9): 47-53.
[3] LI Yaru, WANG Qianqian, CHE Chao, ZHU Deheng. Graph-based Compound-Protein Interaction Prediction with Drug Substructures and Protein 3D Information [J]. Computer Science, 2025, 52(9): 71-79.
[4] LUO Chi, LU Lingyun, LIU Fei. Partial Differential Equation Solving Method Based on Locally Enhanced Fourier NeuralOperators [J]. Computer Science, 2025, 52(9): 144-151.
[5] LIU Leyuan, CHEN Gege, WU Wei, WANG Yong, ZHOU Fan. Survey of Data Classification and Grading Studies [J]. Computer Science, 2025, 52(9): 195-211.
[6] LIU Wei, XU Yong, FANG Juan, LI Cheng, ZHU Yujun, FANG Qun, HE Xin. Multimodal Air-writing Gesture Recognition Based on Radar-Vision Fusion [J]. Computer Science, 2025, 52(9): 259-268.
[7] LIU Zhengyu, ZHANG Fan, QI Xiaofeng, GAO Yanzhao, SONG Yijing, FAN Wang. Review of Research on Deep Learning Compiler [J]. Computer Science, 2025, 52(8): 29-44.
[8] TANG Boyuan, LI Qi. Review on Application of Spatial-Temporal Graph Neural Network in PM2.5 ConcentrationForecasting [J]. Computer Science, 2025, 52(8): 71-85.
[9] ZHENG Cheng, YANG Nan. Aspect-based Sentiment Analysis Based on Syntax,Semantics and Affective Knowledge [J]. Computer Science, 2025, 52(7): 218-225.
[10] FAN Xing, ZHOU Xiaohang, ZHANG Ning. Review on Methods and Applications of Short Text Similarity Measurement in Social Media Platforms [J]. Computer Science, 2025, 52(6A): 240400206-8.
[11] YANG Jixiang, JIANG Huiping, WANG Sen, MA Xuan. Research Progress and Challenges in Forest Fire Risk Prediction [J]. Computer Science, 2025, 52(6A): 240400177-8.
[12] WANG Jiamin, WU Wenhong, NIU Hengmao, SHI Bao, WU Nier, HAO Xu, ZHANG Chao, FU Rongsheng. Review of Concrete Defect Detection Methods Based on Deep Learning [J]. Computer Science, 2025, 52(6A): 240900137-12.
[13] HAO Xu, WU Wenhong, NIU Hengmao, SHI Bao, WU Nier, WANG Jiamin, CHU Hongkun. Survey of Man-Machine Distance Detection Method in Construction Site [J]. Computer Science, 2025, 52(6A): 240700098-10.
[14] CHEN Shijia, YE Jianyuan, GONG Xuan, ZENG Kang, NI Pengcheng. Aircraft Landing Gear Safety Pin Detection Algorithm Based on Improved YOlOv5s [J]. Computer Science, 2025, 52(6A): 240400189-7.
[15] GAO Junyi, ZHANG Wei, LI Zelin. YOLO-BFEPS:Efficient Attention-enhanced Cross-scale YOLOv10 Fire Detection Model [J]. Computer Science, 2025, 52(6A): 240800134-9.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!