计算机科学 ›› 2026, Vol. 53 ›› Issue (9): 333-339.doi: 10.11896/jsjkx.250700144
张虎, 徐国龙, 王宇杰
ZHANG Hu, XU Guolong, WANG Yujie
摘要： 知识型视觉问答(Knowledge-Based Visual Question Answering,KBVQA)旨在通过整合外部知识解答与图像相关的问题,复杂的问题通常需要跨多个知识事实进行多跳推理。然而,现有方法在多跳推理过程中难以有效融合知识事实的全局语义和局部结构特征。为了解决上述问题,提出了超图卷积Transformer(HGCT)框架,该框架结合了卷积神经网络的局部结构建模能力和Transformer的全局上下文感知能力。首先,HGCT框架通过超图结构对视觉实体、问题和知识事实之间的高阶关系进行建模;其次,借助Transformer的注意力机制整合跨模态的全局语义信息,同时利用超边卷积捕获知识超图的局部结构特征,并通过层次化门控网络进行融合;最后,在此基础上设计知识精炼模块,对知识事实的重要性进行评估并加权聚合,结合问题语义输出答案,有效降低了无关知识的影响。在KBVQA数据集KVQA及多跳问答数据集PathQuestion和PathQuestion-Large上的实验结果,展示了所提方法在复杂多跳推理任务中的有效性。
中图分类号:
| [1] YU J,ZHU Z H,WANG Y J,et al.Cross-modal knowledge reasoning for knowledge-based visual question answering[J].Pattern Recognition,2020,108:107563. [2] HEO Y J,KIM E S,CHOI W S,et al.Hypergraph Transfor-mer:Weakly-Supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering[C] //Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.ACL,2022:373-390. [3] WANG P,WU Q,SHEN C H,et al.Explicit Knowledge-based Reasoning for Visual Question Answering[C] //Proceedings of the 26th International Joint Conference on Artificial Intelligence.Melbourne:Morgan Kaufmann,2017:1290-1296. [4] KIM J H,JUN J,ZHANG B T.Bilinear Attention Networks[C] //32nd Conference on Neural Information Processing Systems.MIT Press,2018:1571-1581. [5] WANG Y N,YASUNAGA M,REN H Y,et al.VQA-GNN:Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering[C] //IEEE/CVF International Conference on Computer Vision(ICCV).IEEE,2023:21525-21535. [6] KIM E S,KANG W Y,ON K W,et al.Hypergraph Attention Networks for Multimodal Learning[C] //IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2020:14569-14578. [7] SUN Y H,ZHU Z F,ZUO Z C,et al.DSAMR:Dual-Stream Attention Multi-hop Reasoning for knowledge-based visual question answering[J].Expert Systems with Applications,2024,245:123092. [8] WANG B,XU F Y,LIU P Y,et al.HyperMR:Hyperbolic Hypergraph Multi-hop Reasoning for Knowledge-based Visual Question Answering[C] //Proceedings of the 2024 Joint International Conference on Computational Linguistics,Language Resources and Evaluation(LREC-COLING 2024).Torino:ELRA and ICCL,2024:8505-8515. [9] LING B Y,CHEN X Y,CHEN J,et al.KagNet:Knowledge-Aware Graph Networks for Commonsense Reasoning[C] //2019 Conference on Empirical Methods in Natural Language Proces-sing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP 2019).ACL,2019:2829-2839. [10] SAXENA A,TRIPATHI A,TALUKDAR P.Improving Multi-hop Question Answering over Knowledge Graphs using Know-ledge Base Embeddings[C] //58th Annual Meeting of the Association for Computational Linguistics(ACL 2020).ACL,2020:4498-4507. [11] FENG Y F,TRIPATHI A,YOU H X,et al.Hypergraph Neural Networks[C] //33rd AAAI Conference on Artificial Intelligence/31st Innovative Applications of Artificial Intelligence Conference/ 9th AAAI Symposium on Educational Advances in Artificial Intelligence.AAAI,2019:3558-3565. [12] HAN J L,CHENG B,WANG X.Two-Phase Hypergraph Based Reasoning with Dynamic Relations for Multi-Hop KBQA[C] //Proceedings of the 29th International Joint Conference on Artificial Intelligence.Yokohama:Morgan Kaufmann,2020:3615-3621. [13] YADATI N,DAYANIDHI R,VAISHNAVI S,et al.Knowledge Base Question Answering through Recursive Hypergraphs[C] //16th Conference of the European-Chapter-of-the-Association-for-Computational-Linguistics(EACL).ACL,2021:448-454. [14] CHEN Z R,WANG X,WANG C X,et al.Explainable Link Prediction in Knowledge Hypergraphs[C] //31st ACM Internatio-nal Conference on Information and Knowledge Management(CIKM).ACM,2022:262-271. [15] SHAH S,MISHRA A,YADATI N,et al.KVQA:Knowledge-Aware Visual Question Answering[C] //33rd AAAI Conference on Artificial Intelligence/31st Innovative Applications of Artificial Intelligence Conference/9th AAAI Symposium on Educational Advances in Artificial Intelligence.AAAI,2018:8876-8884. [16] ZHOU M T,HUANG M L,ZHU X Y.An interpretable reasoning network for multi-relation question answering[C] //Proceedings of the 27th International Conference on Computational Linguistics.ACL,2018:2010-2022. [17] KIPF T N,WELLING M.Semi-Supervised Classification withGraph Convolutional Networks[C] //International Conference on Learning Representations(ICLR).OpenReview.net,2017. [18] LI Y J,TARLOW D,BROCKSCHMIDT M,et al.Gated Graph Sequence Neural Networks[C] //International Conference on Learning Representations(ICLR).OpenReview.net,2016. [19] SUKHBAATAR S,SZLAM A,WESTON J,et al.End-To-End Memory Networks[C] //29th Annual Conference on Neural Information Processing Systems(NIPS).MIT Press,2015:2440-2448. [20] SUTSKEVER I,VINYALS O,LE Q,et al.Sequence to Sequence Learning with Neural Networks[C] //28th Conference on Neural Information Processing Systems(NIPS).MIT Press,2014:3104-3112. [21] BORDES A,WESTON J,USUNIER N.Open question answe-ring with weakly supervised embedding models[C] //Joint European Conference on Machine Learning and Knowledge Discovery in Databases.Springer,2014:165-180. [22] BORDES A,CHOPRA S,WESTON J.Question answering with subgraph embeddings[C] //Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP).ACL,2014:615-620. [23] DAS R,DHULIAWALA S,ZAHEER M,et al.Go for a walk and arrive at the answer:Reasoning over paths in knowledge bases using reinforcement learning[C] //International Conference on Learning Representations(ICLR).Vancouver:OpenReview.net,2018. [24] QIU Y Q,WANG Y Z,JIN X L,et al.Stepwise Reasoning for Multi-Relation Question Answering over Knowledge Graph with Weak Supervision[C] //13th Annual ACM International Confe-rence on Web Search and Data Mining(WSDM).ACM,2020:474-482. |
|
||
首页