Abstract: Med-VQA aims to accurately answer clinical questions based on a given medical image,which is key in advancing clinical medical intelligence.Despite some progress in this field,challenges remain in extracting deep multimodal information from both images and questions and in effectively training models on small-scale datasets.To address these issues,this paper proposes a Med-VQA network that incorporates dual data and knowledge enhancement.Aiming at small-scale datasets,a multimodal conditional mixing module is designed to enhance the input image and question data,and linear combinations of input sample pairs are performed by using the category of questions as constraints to improve the rationality of answer generation.For multimodal feature extraction,an image location recognizer based on convolutional neural networks is designed to encode the captured image location features into the fusion process of image and question features for knowledge enhancement,which can effectively achieve feature extraction under fewer parameters.Experimental results on the SLAKE and VQA-RAD datasets demonstrate that the proposed model significantly outperforms the baseline models.

Key words: Visual question answering, Medical visual question answering, Medical images, Data enhancement, Computer vision