Computer Science ›› 2023, Vol. 50 ›› Issue (1): 243-252.doi: 10.11896/jsjkx.220700112

• Artificial Intelligence • Previous Articles     Next Articles

Novel Class Reasoning Model Towards Covered Area in Given Image Based on InformedKnowledge Graph Reasoning and Multi-agent Collaboration

RONG Huan1, QIAN Minfeng2, MA Tinghuai2, SUN Shengjie2   

  1. 1 School of Artificial Intelligence(School of Future Technology),Nanjing University of Information Science and Technology,Nanjing 210044,China
    2 School of Computer Science,Nanjing University of Information Science and Technology,Nanjing 210044,China
  • Received:2022-07-10 Revised:2022-12-06 Online:2023-01-15 Published:2023-01-09
  • About author:RONG Huan,born in 1990,Ph.D,lecturer.His main research interests includesocial media mining,content security on social network,knowledge engineering,etc.
    QIAN Minfeng,born in 2000,undergraduate.His main research interests include pattern recognition,knowledge engineering,etc.
  • Supported by:
    National Natural Science Foundation of China(62102187), Natural Science Foundation of Jiangsu Province(Basic Research Program)(BK20210639) and National Key Research and Development Program of China(2021YFE0104400).

Abstract: Object detection is one of the most popular directions in computer vision,which is widely used in military,medical and other important fields.However,most object detection models can only recognize visible objects.There are often covered(invisible) target objects in pictures in daily life.It is difficult for existing object detection models to show ideal detection performance for covered objects in pictures.Therefore,this paper proposes a novel class reasoning model towards covered area in given image based on informed knowledge graph reasoning and multi-agent collaboration(IMG-KGR-MAC).Specifically,first,IMG-KGR-MAC constructs a global prior knowledge graph according to the visible objects of all pictures in a given picture library and the positional relationship between them.At the same time,according to the objects contained in the pictures themselves and their positional relationships,picture knowledge graphs are established for each picture respectively.The covered objects information in each picture is not included in the global prior knowledge graph and the picture's own knowledge graph.Second,deep deterministic policy gradient(DDPG) deep reinforcement learning idea is adopted to build two cooperative agents.Agent 1 selects the “category label” that is most suitable for the covered object from the global prior knowledge graph according to the semantic information of the current picture,and adds it to the knowledge graph of the given picture as a new entity node.Agent 2 further selects 〈entities,relationships〉 from the global prior knowledge graph according to the newly added entities of agent 1,and expands the graph structure associated with the new entity nodes.Third,agent 1 and agent 2 share the task environment and communicate the reward value,and cooperate with each other to carry out forward and reverse reasoning according to the principles of ‘picture covered target(entity) → associated graph structure' and ‘associated graph structure → picture covered object(entity)',so as to effectively estimate the most likely category label of the covered object of a given picture.Experimental results show that,compared with the existing related methods,the proposed IMG-KGR-MAC model can learn the semantic relationship between the covered picture of a given picture and the global prior knowledge graph,effectively overcome the shortcomings of the existing models that it is difficult to detect the covered object,and has good reasoning ability for the covered object.It has more than 20% improvement in many indicators such as MR(mean rank) and mAP(mean average precision).

Key words: Knowledge graph reasoning, Image object detection, Multi-agent Reinforcement Learning, DDPG

CLC Number: 

  • TP319.1
[1]JIANG S Q,MIN W Q,WANG S Hi.Survey and Prospect of Intelligent Interaction-Oriented Image Recognition Techniques[J].Journal of Computer Research and Development,2016,53(1):113-122.
[2]HARIHARAN B,ARBELÁEZ P,GIRSHICK R,et al.Simultaneous detection and segmentation[C]// European Conference on Computer Vision.Springer,2014:297-312.
[3]HARIHARAN B,ARBELAEZ P,GIRSHICK R,et al.Hypercolumns for object segmentation and fine-grained localization[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2015.
[4]DAI J,HE K,SUN J.Instance-aware semantic segmentation via multi-task network cascades[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:3150-3158.
[5]HE K,GKIOXARI G,DOLLÁR P,et al.Mask r-cnn[C]//2017 IEEE International Conference on Computer Vision(ICCV).IEEE,2017:2980-2988.
[6]KARPATHY A,LI F F.Deep visual-semantic alignments for generating image descriptions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR).2015:3128-3137.
[7]XU K,BA J,KIROS R,et al.Show,attend and tell:Neuralimage caption generation with visual attention[C]//International Conference on Machine Learning,2015:2048-2057.
[8]WU Q,SHEN C,WANG P,et al.Image captioning and visual question answering based on attributes and external knowledge[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(6):1367-1381.
[9]KANG K,LI H,YAN J,et al.T-cnn:Tubelets with convolutional neural networks for object detection fromvideos[C]//IEEE Transactions on Circuits and Systems for Video Techno-logy.2018:2896-2907.
[10]GE Y Z,LIU H,WANG Y,et al.Survey on Deep LearningImage Recognition in Dilemma of Small Samples[J].Journal of Software,2022,33(1):193-210.
[11]CHEN C,QI F.Reviewon Development of Convolutional Neural Network and Its Applicationin Computer Vision[J].Computer Science,2019,46(3):63-73.
[12]ZHANG S,GONG Y H,WANG J J.The Development of Deep Convolution Neural Netwok and Its Applications on Computer Version[J]. Chinese Journal of Computers,2019,42(3):453-482.
[13]CHEN K Q,ZHU Z L,DENG X M,et al.Deep Learning for Multi-Scale Object Detection:A Survey[J].Journal of Software.2021,32(4):1201-1227.
[14]LILLICRAP T P,HUNT J J,PRITZEL A,et al.Continuouscontrol with deep reinforcement learning[C]//ICLR(Poster).2016.
[15]LI Z,JIN X,GUAN S,et al.Path Reasoning over Knowledge Graph:A Multi-agent and Reinforcement Learning Based Me-thod[C]// 2018 IEEE International Conference on Data Mining Workshops(ICDMW).IEEE,2018.
[16]QU M,TANG J.Probabilistic logic neural networks for reaso-ning[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems.2019:7712-7722.
[17]FU C,CHEN T,QU M,et al.Collaborative Policy Learning for Open Knowledge Graph Reasoning[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2672-2681.
[18]LIN X V,SOCHER R,XIONG C.Multi-Hop Knowledge Graph Reasoning with Reward Shaping[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proces-sing.2018:3243-3253.
[19]XU H,JIANG C,LIANG X,et al.Spatial-aware graph relation network for large-scale object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:9298-9307.
[20]FANG Y,KUAN K,LIN J,et al.Object detection meets knowledge graphs[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence.2017:1661-1667.
[21]XU H,JIANG C,LIANG X,et al.Reasoning-RCNN:UnifyingAdaptive Global Reasoning Into Large-Scale Object Detection[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2019.
[22]MARINO K,SALAKHUTDINOV R,GUPTA A.The MoreYou Know:Using Knowledge Graphs for Image Classification[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2017:20-28.
[23]JIANG C,XU H,LIANG X,et al.Hybrid knowledge routedmodules for large-scale object detection[C]//Proceedings of the 32nd International Conference on Neural Information Proces-sing Systems.2018:1559-1570.
[24]WELLING M,KIPF T N.Semi-supervised classification withgraph convolutional networks[C]//International Conference on Learning Representations(ICLR 2017).2016.
[25]DAS R,DHULIAWALA S,ZAHEER M,et al.Go for a Walk and Arrive at the Answer:Reasoning Over Paths in Knowledge Bases using Reinforcement Learning[C]//International Confe-rence on Learning Representations.2018:1-18.
[26]ANG B,YIH S W,HE X,et al.Embedding Entities and Relations for Learning and Inference in Knowledge Bases[C]//Proceedings of the International Conference on Learning Representations(ICLR).2015:1-12.
[27]TROUILLON T,WELBL J,RIEDEL S,et al.Complex embeddings for simple link prediction[C]//International Conference on Machine Learning.PMLR,2016:2071-2080.
[28]VELIKOVI P,CUCURULL G,CASANOVA A,et al.GraphAttention Networks[C]// International Conference on Lear-ning Representations.2018.
[1] SHI Dian-xi, ZHAO Chen-ran, ZHANG Yao-wen, YANG Shao-wu, ZHANG Yong-jun. Adaptive Reward Method for End-to-End Cooperation Based on Multi-agent Reinforcement Learning [J]. Computer Science, 2022, 49(8): 247-256.
[2] DU Wei, DING Shi-fei. Overview on Multi-agent Reinforcement Learning [J]. Computer Science, 2019, 46(8): 1-8.
[3] LIU Dong, LI Su and CAO Zhi-dong. State-of-the-art on Deep Learning and its Application in Image Object Classification and Detection [J]. Computer Science, 2016, 43(12): 13-23.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!