Abstract: Most of the previous multimodal image-text sentiment analysis models use different encoder structures to encode the features of images and text respectively,focusing on exploring different modal feature fusion methods to realize sentiment analysis.However,due to the differences in semantic space between independently extracted features,the semantic associations and complementarities between different features cannot be effectively captured during interaction,which reduces the accuracy of sentiment analysis in turn.To address the above problems,this paper proposes a multimodal image-text sentiment analysis method with multi-view CLIP and hybrid contrast learning.Specifically,the multi-view CLIP feature encoding module employs CLIP to jointly encode image and text representations to improve the semantic consistency of features,and performs multimodal sentiment analysis from multiple perspectives,including image,text,and image-text interaction.In addition,the hybrid contrastive learning module enables the model to extract features with more emotional characteristics and effective information to improve the robustness of the model.In order to remove redundant information in image-text interaction,this paper adopts the fusion strategy of CNN and Transformer cascade,which makes full use of local and global information of image-text to improve the feature representation capability.Finally,comprehensive experiments on three public datasets verify the superiority of the proposed method,and the ablation experiments prove the effectiveness of the components of the proposed method.

Key words: Multimodal, CLIP, Contrastive learning, Pre-trained models, Sentiment analysis