Abstract: Skin melanoma is a kind of disease that can be cured by early detection.The main diagnosis method is based on the manual visual observation of dermatoscope.Affected by the doctor's medical skill and experience,the diagnostic accuracy is 75%~80% and the diagnostic efficiency is low.In this paper,a multi-modal neural network algorithm based on metadata and image data is proposed.Metadata is the feature vector that extracts the basic information of patients,the location of lesions,the resolution and quantity of images through perceptual machine learning model.The image data is extracted from the feature vectors of CNN model,and the two feature vectors are fused and mapped to obtain the disease classification results,which can be used for early auxiliary diagnosis of melanoma.A total of 58 457 samples are collected from ISIC 2019 and ISIC 2020 mixed data sets.The training samples and test samples are divided according to the ratio of 4∶1.The multi-modal algorithm and convolutional neural network method proposed in this paper are used for comparative experimental research.The results show that the AUC value of the melanoma auxiliary diagnosis classification model constructed by this algorithm can be improved by about 1%,which has certain use value.

Key words: Auxiliary diagnosis, Deep learning, Dermoscopy, Melanoma, Metadata, Multimodal algorithm