Abstract: KBVQA(Knowledge-Based Visual Question Answering) aims to answer image-related questions by integrating external knowledge,with complex questions typically requiring multi-hop reasoning across multiple knowledge facts.However,existing methods struggle to effectively fuse the global semantics and local structural features of these knowledge facts during multi-hop reasoning.To address this issue,this paper proposes a hypergraph convolutional Transformer(HGCT) framework,which combines the local structural modeling capability of convolutional neural networks(CNNs) with the global context awareness of Transformers.Firstly,the HGCT framework models high-order relationships among visual entities,questions,and knowledge facts through a hypergraph structure.Secondly,it leverages the Transformer’s attention mechanism to integrate cross-modal global semantic information,while using hyperedge convolution to capture local structural features of the knowledge hypergraph,with fusion realized via a hierarchical gated network.Finally,a knowledge refinement module is designed based on this approach to evaluate the importance of knowledge facts,perform weighted aggregation,and combine question semantics to output answers,thereby effectively reducing the impact of irrelevant knowledge.Experimental results on the KBVQA dataset KVQA,as well as the multi-hop question answering datasets PathQuestion and PathQuestion-Large,demonstrate the effectiveness of the proposed method in complex multi-hop reasoning tasks.

Key words: Visual question answering, Knowledge graph, Multi-hop reasoning, Hypergraph, Feature fusion