Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (10): 104-111.doi: 10.11896/jsjkx.221000084

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Unbiased Scene Graph Generation Based on Adaptive Regularization Algorithm

LI Haochen1, CAO Fuyuan1,2, QIAO Shichang1   

  1. 1 School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China
    2 Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education,Shanxi University,Taiyuan 030006,China
  • Received:2022-10-11 Revised:2023-03-07 Online:2023-10-10 Published:2023-10-10
  • About author:LI Haochen,born in 1995,postgra-duate.His main research interests include computer vision and data mining.CAO Fuyuan,born in 1974,professor,Ph.D supervisor,is a senior member of China Computer Federation.His main research interests include data mining and machine learning.
  • Supported by:
    National Natural Science Foundation of China(61976128) and Applied Basic Research Program of Shanxi Pro-vince(201901D111035).

PDF (PC)

1433

Abstract: The purpose of scene graph generation is to give a picture,obtain the visual triplet form of entities and relationships between entities through the object detection module,namely subject,relationship and object,and construct a semantic structured representation.Scene graphs can be applied to downstream tasks such as image retrieval and visual question answering.However,due to the longtail distribution of relationships between entities in the dataset,existing models tend to predict coarse grained head relationships.Such scene graph cannot play an auxiliary role for downstream tasks.Previous works generally adopt rebalancing strategies such as resampling and reweighting to solve the long tail problem.However,because the models repeatedly learn the tail relationship samples,it is prone to overfitting.In order to solve the above problems,an adaptive regularized unbiased scene graph generation method is proposed in this paper.Specifically,the method adaptively adjusts the weights of full connected classifier of the model by designing a regularization term based on the prior relation frequency,so as to achieve the prediction of model balance.The proposedmethod is tested on Visual Genome dataset,and the experimental results show that it can not only prevent the model from overfitting,but also alleviate the negative impact of the longtail distribution problem on the scene graph generation,and the state-of-the-artscene graph generation methods combined with the proposed method can more effectively improve the performance of unbiased scene graph generation.

Key words: Scene graph, Long-tail distribution, Re-sampling, Re-weighting, Adaptive regularization

CLC Number: 

  • TP391
[1]JOHNSON J,KRISHNA R,STARK M,et al.Image Retrieval Using Scene Graph[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2015:3668-3678.
[2]GUO Y,CHEN J J,ZHANG H,et al.Visual Relations Aug-mented Cross-modal Retrieval[C]//Proceedings of ICMR International Conference on Multimedia Retrieval.2020.
[3]SONG X,CHEN J J,WU Z X,et al.Spatial-Temporal Graphs for Cross-Modal Text2Video Retrieval[J].IEEE Transactions on Multimedia,2021,24:2914-2923.
[4]TANG K H,ZHANG H,WU B,et al.Learning to Compose Dynamic Tree Structures for Visual Contexts[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:6619-6628.
[5]HILDEBRANDT M,LI H,KONER R,et al.Scene Graph Reasoning for Visual Question Answering[J].arXiv:2007.01072,2020.
[6]CHEN S Z,JIN Q,WANG P,et al.Say as You Wish:Fine-grained Control of Image Caption Generation with Abstract Scene Graphs[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2020:9962-9971.
[7]GU J X,JOTY S,CAI J F,et al.Unpaired Image Captioning Via Scene Graph Alignments[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:10323-10332.
[8]GUO Y Y,GAO L L,WANG X H,et al.From General to Specific:Informative Scene Graph Generation Via Balance Adjustment[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2021.
[9]KANG B,XIE S,ROHRBACH M,et al.Decoupling Representation and Classifier for Long-Tailed Recognition[J].arXiv:1910.09217,2019.
[10]DAI B,ZHANG Y Q,LIN D H,et al.Detecting Visual Relationships with Deep Relational Networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2017:3076-3086.
[11]LIANG K M,GUO Y H,CHANG H,et al.Visual Relationship Detection with Deep Structural Ranking[C]//Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence.2018.
[12]LIAO W T,ROSENHAHN B,SHUAI L,et al.Natural Lan-guage Guided Visual Relationship Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019.
[13]LI R J,ZHANG S Y,WAN B,et al.Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Ge-neration[C]//Proceedings of IEEEConference on Computer Vision and Pattern Recognition.New York:IEEE Press,2021:11109-11119.
[14]XU D F,ZHU Y K,CHOY C B,et al.Scene Graph Generation by Iterative Message Passing[C]//Proceedings of IEEE Confe-rence on Computer Vision and Pattern Recognition.New York:IEEE Press,2017:5410-5419.
[15]ZELLERS R,YATSKAR M,THOMSON S,et al.Neural Motifs:Scene Graph Parsing with Global Context[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:5831-5840.
[16]YANG J W,LU J S,LEE S,et al.Graph R-CNN for SceneGraph Generation[C]//Proceedings of the ECCV European Conference on Computer Vision.2018:670-685.
[17]CHEN T S,YU W H,CHEN R Q,et al.Knowledge-embedded Routing Network for Scene Graph Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:6163-6171.
[18]HUANG Y T,YAN H.Scene Graph Generation Model Combi-ning Attention Mechanism and Feature Fusion [J].Chinese Computer Science,2020,47(6):113-137.
[19]LU Y C,RAI H,CHANG J,et al.Context-Aware Scene Graph Generation with Seq2seq Transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:15931-15941.
[20]ZAREIAN A,KARAMAN S,CHANG S F.Bridging knowledge graphs to generate scene graphs[C]//Proceedings of the ECCV European Conference on Computer Vision.Cham:Springer,2020:606-623.
[21]TANG K H,NIU Y L,HUANG J Q,et al.Unbiased Scene Graph Generation from Biased Training[C]//Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition.2020:3716-3725.
[22]KRISHNA R,ZHU Y,GROTH O,et al.Visual Genome:Connecting Language and Vision Using Crowdsourced Dense Image Annotations[J].International Journal of Computer Vision,2017,123(1):32-73.
[23]LYU X,GAO L,GUO Y,et al.Fine-Grained Predicates Lear-ning for Scene Graph Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:19467-19475.
[24]CHIOU M J,DING H,YAN H,et al.Re-covering the Unbiased Scene Graphs from t-he biased ones[C]//Proceedings of the 29th ACM International Conference on Multimedia.2021:1581-1590.
[25]LU C,KRISHNA R,BERNSTEIN M,et al.Visual Relationship Detection with Language Priors[C]//Proceedings of the ECCV European Conference on Computer Vision.Cham:Springer,2016:852-869.
[26]CHEN C,ZHAN Y,YU B,et al.Resistance Training usingPrior Bias:toward Unbiased Scene Graph Generation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2022:212-220.
[27]LIN X,DING C,ZENG J,et al.Gps-net:GraphProperty Sensing Network for Scene Graph Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:3746-3753.
[28]YAN S,SHEN C,JIN Z,et al.Pcpl:Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:265-273.
[29]SUHAIL M,MITTAL A,SIDDIQUIE B,et al.Energy-basedLearning for Scene Graph Generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:13936-13945.
[30]YU J,CHAI Y,WANG Y,et al.Cogtree:Cognition Tree Loss for Unbiased Scene Gr-aph generation[J].arXiv:2009.07526,2020.
[1] HUANG Yong-tao, YAN Hua. Scene Graph Generation Model Combining Attention Mechanism and Feature Fusion [J]. Computer Science, 2020, 47(6): 133-137.
[2] ZHUANG Zhi-gang, XU Qing-lin. Scene Graph Generation Model Combining Multi-scale Feature Map and Ring-type RelationshipReasoning [J]. Computer Science, 2020, 47(4): 136-141.
[3] LI Zhi and XIE Qiang. Moving Target Tracking Based on Improved Particle Filter [J]. Computer Science, 2014, 41(2): 232-235.
[4] ZHAO Na,GUO Li,YUAN Hong-xing. Research on Free Modeling Method Based on Free Hand Sketching [J]. Computer Science, 2010, 37(12): 262-265.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.