Abstract: Scene graph generation(SGG) methods play a pivotal role in studying objects and their relationships within images,with widespread applications in visual understanding and image retrieval.However,existing SGG methods are limited by visual features or individual visual concepts such as objects,resulting in a low accuracy of relationship recognition and necessitating a substantial amount of manual annotation.To address the aforementioned issues,this paper integrates image and text features and proposes a multimodal contrastive learning based scene graph generation method,multimodal contrastive learning for scene graph(MCL-SG).This method begins by extracting features from both image and text inputs,obtaining image and text features.Subsequently,a Transformer Encoder is employed to encode and fuse feature vectors,enabling a synergistic integration of information from diverse sources.Notably,MCL-SG incorporates a self-supervised contrastive learning strategy,calculating the similarity between image and text features.Training is accomplished by minimizing the dissimilarity between positive and negative samples,eliminating the need for extensive manual annotation.In this study,experiments are conducted using the VG(Visual Genome) dataset,a substantial public dataset for scene graph generation.Experiments are structured into three distinct hierarchical subtasks:SGDet,SGCls,and PredCls and the results demonstrate that,in the mean Recall@100 metric,MCL-SG achieves a 9.8% improvement in scene graph detection,a significant 14.0% enhancement in scene graph classification,and an 8.9% boost in relationship classification,thus proving the effectiveness of MCL-SG.

Key words: Scene graph generation, Transformer model, Multimodal, Contrastive learning, Object detection