Computer Science ›› 2022, Vol. 49 ›› Issue (5): 105-112.doi: 10.11896/jsjkx.210100108

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Fine-grained Image Classification Based on Multi-branch Attention-augmentation

ZHANG Wen-xuan, WU Qin   

  1. School of Artificial Intelligence and Computer Science,Jiangnan University,Jiangsu,Wuxi 214122
    ChinaJiangsu Provincial Engineering Laboratory for Pattern Recognition and Computational Intelligence,Jiangnan University,Jiangsu,Wuxi 214122,China
  • Received:2021-01-14 Revised:2021-04-21 Online:2022-05-15 Published:2022-05-06
  • About author:ZHANG Wen-xuan,born in 1997,master candidate,is a member of China Computer Federation.His main research interests include computer vision and machine learning.
    WU Qin,born in 1978,Ph.D,associate professor,is a member of China Computer Federation.Her main research interests include computer vision and pattern recognition.
  • Supported by:
    National Natural Science Foundation of China(61972180).

Abstract: In order to address the challenges of high intra-class variances and low inter-class variances in fine-grained image classification,a multi-branch attention-augmented convolution neural network is proposed to solve the problem.The pre-trained Inception-V3 network is used to extract basic feature.In order to solve the problem that features are extracted from one part of an object and encourage the network to pay more attention to the discriminative features of different parts,we apply self-constrained attention-wised cropping and self-constrained attention-wised erasing on the central parts of the original images.It also improves the detection accuracy of object locations.Meanwhile,a central regularization loss function is proposed to constrain attention-augmented training process to obtain better attention regions and expand the gap between different classes of images.Comprehensive experiments on three benchmark datasets show that our approach surpasses the state-of-art works.

Key words: Fine-grained image classification, Weakly supervised learning, Multi-branch attention-augmentation, Convolutional neural network, Central regularization loss

CLC Number: 

  • TP391
[1]WELINDER P,BRANSON S,MITA T,et al.The Caltech-UCSD Birds-200-2011 Dataset[R].California Institute of Technology,2011:1-15.
[2]RABIEE H,HADDADNIA J,MOUSAVI H,et al.Novel dataset for fine-grained abnormal behavior understanding in crowd[C]//IEEE International Conference on Advanced Video & Signal Based Surveillance.2016:121-130.
[3]YANG W G,HUAI Y J.Flower Image Enhancement and Classification Based on Deep Convolution Generative Adversarial Network[J].Computer Science,2020,47(6):176-179.
[4]KRAUSE J,STARK M,DENG J,et al.3D Object Representations for Fine-Grained Categorization[C]//IEEE International Conference on Computer Vision Workshops.2013:554-561.
[5]MAJI S,RAHTU E,KANNALA J,et al.Fine-Grained VisualClassification of Aircraft[C]//IEEE International Conference on Advanced Video & Signal Based Surveillance.2013:1-6.
[6]PERRONNIN F,DANCE C.Fisher Kernels on Visual Vocabularies for Image Categorization[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition.2007:1-8.
[7]SÁNCHEZ J,MENSINK T,VERBEEK J.Image Classification with the Fisher Vector:Theory and Practice[J].International Journal of Computer Vision,2013,105(1):222-245.
[8]LOWE D G.Object recognition from local scale-invariant fea-tures[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision.1999:1150-1157.
[9]DALAL N,TRIGGS B.Histograms of Oriented Gradients for Human Detection[C]//IEEE Computer Society Conference on Computer Vision & Pattern Recognition.2005.
[10]DONAHUE J,JIA Y Q,VINYALS O,et al.DeCAF:A Deep Convolutional Activation Feature for Generic Visual Recognition[C]//Proceedings of the 31st International Conference on Machine Learning.PMLR,2014:647-655.
[11]HE K,ZHANG X,REN S,et al.Deep Residual Learning forImage Recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:770-778.
[12]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//ICLR.2015:1-14.
[13]SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the Inception Architecture for Computer Vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2016:2818-2826.
[14]XIE L,HUANG C.A Residual Network of Water Scene Recognition Based on Optimized Inception Module and Convolutional Block Attention Module[C]//2019 6th International Conference on Systems and Informatics (ICSAI).2019:1174-1178.
[15]SUN G,CHOLAKKAL H,KHAN S,et al.Fine-Grained Recognition:Accounting for Subtle Differences between Similar Classes[J].Proceedings of the AAAI Conference on Artificial Intelligence,2020,34(1):12047-12054.
[16]TAN M,WANG G,ZHOU J,et al.Fine-Grained Classification via Hierarchical Bilinear Pooling With Aggregated Slack Mask[J].IEEE Access,2017,7(1):117944-117953.
[17]YAO B,BRADSKI G,LI F F.A codebook-free and annotation-free approach for fine-grained image categorization[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition.2012:3466-3473.
[18]CHERIYADAT A M.Unsupervised Feature Learning for Aerial Scene Classification[J].IEEE Transactions on Geoscience and Remote Sensing,2014,52(1):439-451.
[19]ZHANG N,DONAHUE J,GIRSHICK R,et al.Part-based R-CNNs for Fine-grained Category Detection[C]//European Conference on Computer Vision(ECCV).2014:834-849.
[20]HE K,GKIOXARI G,DOLLÁR P,et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,42(2):386-397.
[21]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//2015 IEEE Confe-rence on Computer Vision and Pattern Recognition (CVPR).2015:3431-3440.
[22]GE W,LIN X,YU Y.Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification From the Bottom Up[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:3029-3038.
[23]XIAO T J,XU Y C,YANG K Y,et al.The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2015:842-850.
[24]LIN T,ROYCHOWDHURY A,MAJI S.Bilinear CNN Models for Fine-Grained Visual Recognition[C]//2015 IEEE International Conference on Computer Vision (ICCV).2015:1449-1457.
[25]ZHOU M,BAI Y,ZHANG W,et al.Look-Into-Object:Self-Supervised Structure Modeling for Object Recognition[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2020:11771-11780.
[26]LIU C,XIE H,ZHA Z J,et al.Filtration and Distillation:Enhancing Region Attention for Fine-Grained Visual Categorization[C]//AAAI Conference on Artificial Intelligence.2020:11555-11562.
[27]HUANG S,WANG X,DAO D.SnapMix:Semantically Proportional Mixing for Augmenting Fine-grained Data[C]//AAAI Conference on Artificial Intelligence.2021:1-8.
[28]WU J,XU J,DING T.Fine-grained Image Classification Algorithm Based on Ensemble Methods of Transfer Learning[J].Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition),2020,32(3):452-458.
[29]ZHENG H,FU J,MEI T,et al.Learning Multi-attention Con-volutional Neural Network for Fine-Grained Image Recognition[C]//2017 IEEE International Conference on Computer Vision (ICCV).2017:5219-5227.
[30]SUN M,YUAN Y,ZHOU F,et al.Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition[C]//European Conference on Computer Vision(ECCV).2018:834-850.
[31]YANG Z,LUO T,WANG D,et al.Springer International Publishing Learning to Navigate for Fine-Grained Classification[C]//European Conference on Computer Vision(ECCV).2018:438-454.
[32]LUO W,ZHANG H,LI J,et al.Learning Semantically En-hanced Feature for Fine-Grained Image Classification[J].IEEE Signal Processing Letters,2020,27:1545-1549.
[33]CHEN Y,BAI Y,ZHANG W,et al.Destruction and Construction Learning for Fine-Grained Image Recognition[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:5152-5161.
[34]HU T,QI H.See Better Before Looking Closer:Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification[J/OL].
[35]ZHAO B,WU X,FENG J,et al.Diversified Visual Attention Networks for Fine-Grained Object Classification[J].IEEE Transactions on Multimedia,2017,19(6):1245-1256.
[36]DUBEY A,GUPTA O,GUO P,et al.Pairwise Confusion for Fine-Grained Visual Classification[C]//European Conference on Computer Vision(ECCV).2018:71-88.
[1] HU Fu-yuan, WAN Xin-jun, SHEN Ming-fei, XU Jiang-lang, YAO Rui, TAO Zhong-ben. Survey Progress on Image Instance Segmentation Methods of Deep Convolutional Neural Network [J]. Computer Science, 2022, 49(5): 10-24.
[2] XU Hua-chi, SHI Dian-xi, CUI Yu-ning, JING Luo-xi, LIU Cong. Time Information Integration Network for Event Cameras [J]. Computer Science, 2022, 49(5): 43-49.
[3] ZHAO Ren-xing, XU Pin-jie, LIU Yao. ECG-based Atrial Fibrillation Detection Based on Deep Convolutional Residual Neural Network [J]. Computer Science, 2022, 49(5): 186-193.
[4] LI Zi-yi, ZHOU Xia-bing, WANG Zhong-qing, ZHANG Min. Stance Detection Based on User Connection [J]. Computer Science, 2022, 49(5): 221-226.
[5] LIU Lin-yun, CHEN Kai-yan, LI Xiong-wei, ZHANG Yang, XIE Fang-fang. Overview of Side Channel Analysis Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(5): 296-302.
[6] GAO Yue, FU Xiang-ling, OUYANG Tian-xiong, CHEN Song-ling, YAN Chen-wei. EEG Emotion Recognition Based on Spatiotemporal Self-Adaptive Graph ConvolutionalNeural Network [J]. Computer Science, 2022, 49(4): 30-36.
[7] ZHAO Kai, AN Wei-chao, ZHANG Xiao-yu, WANG Bin, ZHANG Shan, XIANG Jie. Intracerebral Hemorrhage Image Segmentation and Classification Based on Multi-taskLearning of Shared Shallow Parameters [J]. Computer Science, 2022, 49(4): 203-208.
[8] XU Hua-jie, QIN Yuan-zhuo, YANG Yang. Scene Recognition Method Based on Multi-level Feature Fusion and Attention Module [J]. Computer Science, 2022, 49(4): 209-214.
[9] LI Guo-quan, YAO Kai, PANG Yu. Automatic Identification Algorithm of Blood Cell Image Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(4): 247-253.
[10] LIU Shuo, WANG Geng-run, PENG Jian-hua, LI Ke. Chinese Short Text Classification Algorithm Based on Hybrid Features of Characters and Words [J]. Computer Science, 2022, 49(4): 282-287.
[11] ZHOU Ying, CHANG Ming-xin, YE Hong, ZHANG Yan. Super Resolution Reconstruction Method of Solar Panel Defect Images Based on Meta-transfer [J]. Computer Science, 2022, 49(3): 185-191.
[12] ZUO Jie-ge, LIU Xiao-ming, CAI Bing. Outdoor Image Weather Recognition Based on Image Blocks and Feature Fusion [J]. Computer Science, 2022, 49(3): 197-203.
[13] ZHANG Lyu, ZHOU Bo-wen, WU Liang-hong. SSD Network Based on Improved Convolutional Attention Module and Residual Structure [J]. Computer Science, 2022, 49(3): 211-217.
[14] LI Hao, ZHANG Lan, YANG Bing, YANG Hai-xiao, KOU Yong-qi, WANG Fei, KANG Yan. Fine-grained Sentiment Classification of Chinese Microblogs Combining Dual Weight Mechanismand Graph Convolutional Neural Network [J]. Computer Science, 2022, 49(3): 246-254.
[15] LENG Jia-xu, WANG Jia, MO Meng-jing-cheng, CHEN Tai-yue, GAO Xin-bo. Survey on Video Super-resolution Based on Deep Learning [J]. Computer Science, 2022, 49(2): 123-133.
Full text



[1] LIU Bo-yi, TANG Xiang-yan and CHENG Jie-ren. Recognition Method for Corn Borer Based on Templates Matching in Muliple Growth Periods[J]. Computer Science, 2018, 45(4): 106 -111 .
[2] XU Li-li, DONG Yi-hong, PAN Jian-fei and CHEN Hua-hui. Survey of Graph Sparsification Algorithms for Complex Networks[J]. Computer Science, 2018, 45(5): 24 -30 .
[3] LI Rong-yu , ZHANG Wei-jie , ZHOU Zhi-yong. Improved PSO Algorithm and Its Load Distribution Optimization of Hot Strip Mills[J]. Computer Science, 2018, 45(7): 214 -218 .
[4] NAN Ming-li, LI Jian-hua, CUI Qiong, RAN Hao-dan. Flexibility Measurement Model of Command and Control Information Chain for Networked Operations[J]. Computer Science, 2018, 45(10): 306 -312 .
[5] XU Su-ping, YANG Xi-bei, YU Hua-long and YU Dong-jun. Neighborhood Collaborative Representation Based Classification Method[J]. Computer Science, 2017, 44(9): 234 -238 .
[6] HAO Xin and GUO Shao-zhong. Optimization of 3D Finite Difference Algorithm on Intel MIC[J]. Computer Science, 2017, 44(5): 26 -32 .
[7] LIN Xin-hua, QIN Qiang, LI Shuo, WEN Min-hua and MATSUOKA Satoshi. Evaluating Intel AVX2 Vgather Instructions with Stencils[J]. Computer Science, 2017, 44(1): 20 -24 .
[8] FAN Xing-qi, LI Xue-feng, ZHAO Su-yun, CHEN Hong and LI Cui-ping. Weighted Attribute Reduction Based on Fuzzy Rough Sets[J]. Computer Science, 2018, 45(1): 133 -139 .
[9] HU Xian-jun, CHEN Jian-xin, ZHOU Sheng-qiang and CHENG Yi. Study on IEEE 802.15.4 Scheduling Algorithms for Real-time Communication[J]. Computer Science, 2015, 42(Z11): 222 -226 .
[10] LI Qing-sheng, ZHANG Li, LIU Quan, XIONG Jing and YANG Xin-xin. Chinese Character Computing Model Based on Cloud Information Protection[J]. Computer Science, 2015, 42(11): 73 -79 .